Facial Emotion Detection¶

Problem Definition¶

The context: Why is this problem important to solve?
The objectives: What is the intended goal?
The key questions: What are the key questions that need to be answered?
The problem formulation: What are we trying to solve using data science?


Context¶


⏩ Deep Learning has found applications in many predictive tasks relating to more unstructured forms of data over the last few years, such as images, text, audio and video. Many of these tasks seem to be in the vein of a larger direction of predictive modeling that aims to match human-level performance on such tasks, because humans have evolved to specialize in performing intelligent actions on such unstructured data. As a specific branch of AI (also called Affective Computing or Emotion AI) Artificial Emotional Intelligence stands for the study and development of technologies and computers that can read human emotions by means of analyzing body gestures, facial expressions, voice tone, etc. and react appropriately to them.

In the field of human-machine interaction, facial expression recognition is critical. From recent research, it has been found that as much as 55% of communication of sentiment takes place through facial expressions and other visual cues. Therefore, training a model to identify facial emotions accurately is an important step towards the development of emotionally intelligent behavior in machines with AI capabilities. Automatic facial expression recognition systems could have many applications, including but not limited to any use case that requires human behavior understanding, detection of mental disorders, and creating a higher quality of virtual assistant for customer-facing businesses.


Objective¶


⏩ The goal of this project is to use Deep Learning and Artificial Intelligence techniques to create a computer vision model that can accurately detect facial emotions. The model should be able to perform multi-class classification on images of facial expressions, to classify the expressions according to the associated emotion.


About the dataset¶


⏩ The data set consists of 3 folders, i.e., 'test', 'train', and 'validation'. Each of these folders has four subfolders:

‘happy’: Images of people who have happy facial expressions.
‘sad’: Images of people with sad or upset facial expressions.
‘surprise’: Images of people who have shocked or surprised facial expressions.
‘neutral’: Images of people showing no prominent emotion in their facial expression at all.

Mounting the Drive¶

NOTE: Please use Google Colab from your browser for this notebook. Google.colab is NOT a library that can be downloaded locally on your device.

In [1]:
from google.colab import drive
drive.mount('/content/drive')
Mounted at /content/drive
In [2]:
!ls '/content/drive/MyDrive/Colab Notebooks/Capstone/data/'
Facial_emotion_images.zip

Importing the Libraries¶

In [3]:
# Importing data support libraries, including graph support
import math
import matplotlib.pyplot as plt
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import numpy as np
import operator
import os
import pandas as pd
import pickle
import random
import seaborn as sns
import zipfile

# Importing Machine Learning utilities
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
from sklearn.metrics import f1_score
from sklearn.metrics import accuracy_score

# Importing Deep Learning Libraries
import tensorflow as tf
import tensorflow.keras.applications as ap

from tensorflow.keras import backend
from tensorflow.keras.layers import Activation
from tensorflow.keras.layers import BatchNormalization
from tensorflow.keras.layers import Conv2D
from tensorflow.keras.layers import Dense
from tensorflow.keras.layers import Dropout
from tensorflow.keras.layers import Embedding
from tensorflow.keras.layers import Flatten
from tensorflow.keras.layers import GlobalAveragePooling2D
from tensorflow.keras.layers import Input
from tensorflow.keras.layers import LeakyReLU
from tensorflow.keras.layers import LSTM
from tensorflow.keras.layers import MaxPooling2D
from tensorflow.keras.layers import Reshape
from tensorflow.keras.losses import categorical_crossentropy
from tensorflow.keras.models import Model
from tensorflow.keras.models import Sequential
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.optimizers import RMSprop
from tensorflow.keras.optimizers import SGD
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.preprocessing.image import img_to_array
from tensorflow.keras.preprocessing.image import load_img
from tensorflow.keras.utils import to_categorical

from keras.applications.vgg16 import VGG16
from keras.callbacks import EarlyStopping
from keras.callbacks import ModelCheckpoint
from keras.callbacks import ReduceLROnPlateau

# For environment checklist
from psutil import virtual_memory
In [4]:
# Reviewing TensorFlow library version
print('TF version: ', tf.__version__)

try:
  tpu = tf.distribute.cluster_resolver.TPUClusterResolver()  # TPU detection
  print('Running on TPU ', tpu.cluster_spec().as_dict()['worker'])
except:
  print('ERROR: Not connected to a TPU runtime')
TF version:  2.12.0
Running on TPU  ['10.30.87.242:8470']
In [5]:
# Reviewing the memory available
ram_gb = virtual_memory().total / 1e9
print('Runtime has {:.1f} gigabytes of available RAM'.format(ram_gb))

if ram_gb < 20:
  print('Not using a high-RAM runtime')
else:
  print('Using a high-RAM runtime!')
Runtime has 37.8 gigabytes of available RAM
Using a high-RAM runtime!

Let us load and unzip the data¶

Note:

  • You must download the dataset from the link provided on Olympus and upload the same on your Google drive before executing the code in the next cell.
  • In case of any error, please make sure that the path of the file is correct as the path may be different for you.
In [6]:
file = '/content/drive/MyDrive/Colab Notebooks/Capstone/data/Facial_emotion_images.zip'

# The data is provided as a zip file so we need to extract it from the zip file
with zipfile.ZipFile(file, 'r') as zip_ref:
    zip_ref.extractall()

Global Variables¶

In [7]:
# Global variables
IMG_DIR = '/content/Facial_emotion_images'
MODEL_DIR = f'/content/drive/MyDrive/Colab Notebooks/Capstone/model'

TRAIN_PATH = IMG_DIR + '/train'
TEST_PATH = IMG_DIR + '/test'
VAL_PATH = IMG_DIR + '/validation'

SEED = 0

FACE_EXPRESSIONS = ['happy', 'neutral', 'sad', 'surprise']

Visualizing our Classes¶

Let's look at our classes.

Write down your observation for each class. What do you think can be a unique feature of each emotion, that separates it from the remaining classes?

In [8]:
def reading_files(path, name_of_set, show_sample_images=True, pick_random=True):
  """Read files in the directory path.
  It can shows random sample of images if it is desired by setting
  show_sample_images and pick_random parameters.
  """
  files_to_read = os.listdir(path)
  print(f'Total files in {name_of_set}: {len(files_to_read)}')

  if show_sample_images:
    fig = plt.figure(figsize=(10, 2))
    if pick_random:
      images = [os.path.join(path,
                            files_to_read[random.randint(0, len(files_to_read)-1)])
                for i in range(8)]
    else:
      images = [os.path.join(path, file) for file in sorted(files_to_read)[-8:]]

    for i, image in enumerate(images, 1):
        ax = plt.subplot(1, 8, i)
        ax.axis('Off')
        plt.imshow(load_img(image))
        plt.title(image.split('/')[-1], fontsize=6)
    fig.suptitle(name_of_set, fontsize=8)
    plt.tight_layout()
    plt.show()
  return files_to_read
In [9]:
def read_and_display_images_in_set(path, set_name, show_sample_images=True, pick_random=True):
  """Read over different subfolders (categories) in the directory path.
  """
  files_in_set = {}
  expressions = FACE_EXPRESSIONS
  feature = []
  for expression in expressions:
    files_path = f'{path}/{expression}'
    name_of_set = f'{set_name} - {expression.upper()} faces'
    files_in_set[expression] = reading_files(files_path, name_of_set, show_sample_images, pick_random)
    feature += [expression] * len(files_in_set[expression])

  print(f'Total images: {len(feature)}')
  return files_in_set, pd.Series(feature)
In [10]:
def get_feature_countplot(feature, set_name):
  """Countplot of a categorical variable with quantity label on each catego.
  """
  sns.set(font_scale=.75)
  total = len(feature) # Length of the column
  plt.figure(figsize=(10, 3))

  # Convert the column to a categorical data type
  feature = feature.astype('category')
  origin = feature.copy()

  labels = feature.value_counts().index
  ax = sns.countplot(x=feature, palette='Paired', order=labels)
  ax.set_xlabel('')

  # custom label calculates percent and add an empty string so 0 value bars don't have a number
  for container in ax.containers:
    labels = [f'{h:.0f}\n( {h/origin.count()*100:0.1f}% )'
              if (h := v.get_height()) > 0 else '' for v in container]
    ax.bar_label(container, labels=labels, label_type='edge',
                 fontsize='small', weight='bold') # color='white', label_type='center'
    ylim = plt.ylim()
    plt.ylim(ylim[0], ylim[1]*1.1)

    plt.title(f'Data Distribution in {set_name}')
    plt.show()
    plt.style.use('default')

Training Set

In [11]:
title_of_set = 'Train Set'
training_set, feature_train = read_and_display_images_in_set(TRAIN_PATH, title_of_set)
Total files in Train Set - HAPPY faces: 3976
Total files in Train Set - NEUTRAL faces: 3978
Total files in Train Set - SAD faces: 3982
Total files in Train Set - SURPRISE faces: 3173
Total images: 15109
In [12]:
get_feature_countplot(feature_train, title_of_set)

Testing Set

In [13]:
title_of_set = 'Test Set'
testing_set, feature_test = read_and_display_images_in_set(TEST_PATH, title_of_set, pick_random=False)
Total files in Test Set - HAPPY faces: 32
Total files in Test Set - NEUTRAL faces: 32
Total files in Test Set - SAD faces: 32
Total files in Test Set - SURPRISE faces: 32
Total images: 128
In [14]:
get_feature_countplot(feature_test, title_of_set)

Validation Set

In [15]:
title_of_set = 'Validation Set'
validation_set, feature_val = read_and_display_images_in_set(VAL_PATH, title_of_set)
Total files in Validation Set - HAPPY faces: 1825
Total files in Validation Set - NEUTRAL faces: 1216
Total files in Validation Set - SAD faces: 1139
Total files in Validation Set - SURPRISE faces: 797
Total images: 4977
In [16]:
get_feature_countplot(feature_val, title_of_set)

⏩ Observations and Insights:

  • There are 3 different sets: Training, Testing, and Validation.
  • All of them have images for the expressions: Happy, Sad, Surprise, and Neutral.
  • Images are in grey but they are coded in 3 channels. It will be explored how having them encoded in just one channel (grayscale ) will affect the model performance. The initial supposition is that images in grayscale (one channel) will speed the process and make the model more efficient in finding features to classify images. In the end, it will be verified this supposition.
  • There are 15,109 images in the training set. Although the images for the Surprise expressions are fewer (21%) than the other categories (26%), the difference is not significant. Data is not unbalanced.
  • The testing set has few images (128 in total) but they are equally distributed in each expression. However, one image in the testing set (15838.jpg) is not a face. It needs to be removed.
  • The validation set has 4,977 images. The distribution is not uniform, the Happy expression Category has more than double the images that the Surprise expression Category has.
  • The samples are shown randomly, however, it seems that most of the faces, in the training set, are in front positions, meanwhile in the testing set and in the validation set, the majority of faces are tilted. We can try to fix this when we prepare the image loaders.
  • Some images look like they are in the wrong category, for example, image 9984.jpg does not look like a happy face. This image is found in the validation set. This situation is present in all sets: training, testing, and validation. Another image is the 32222.jpg in the training set, which looks like Happy but is in Neutral categories.
  • The images in happy class have some unique features like upturned corners of the lips, and the images in surprise also have some unique features like an open mouth in an "o" shape. Meanwhile, the images in the sad have different features, p.e. corners of the lips turned down, frown. With the neutral class, the most common characteristic is lips in a straight line.
In [17]:
fig = plt.figure(figsize=(6, 2))
ax = plt.subplot(1, 2, 1)
ax.axis('Off')
plt.imshow(load_img(f'{TRAIN_PATH}/neutral/32222.jpg'))
plt.title('Trainning Set \nNeutral \n32222.jpg', fontsize=9)
ax = plt.subplot(1, 2, 2)
ax.axis('Off')
plt.imshow(load_img(f'{VAL_PATH}/happy/9984.jpg'))
plt.title('Validation Set \nHappy \n9984.jpg', fontsize=9)
plt.tight_layout()
plt.show()
In [18]:
# Deleting the wrong images
!rm '/content/Facial_emotion_images/test/happy/15838.jpg'
In [19]:
# Recalculating the testing_set and feature_test
title_of_set = 'Test Set'
testing_set, feature_test = read_and_display_images_in_set(TEST_PATH, title_of_set, show_sample_images=False)
Total files in Test Set - HAPPY faces: 31
Total files in Test Set - NEUTRAL faces: 32
Total files in Test Set - SAD faces: 32
Total files in Test Set - SURPRISE faces: 32
Total images: 127
In [20]:
# Observing the dimenssions of images in each set
def display_dimmension_images_in_set(path, data_set, set_name):
  """Display a scatterplot of images dimmension in the set
  """
  dim = []
  for expression, images in data_set.items():
    files_path = f'{path}/{expression}'
    dim += [(img_to_array(load_img(f'{files_path}/{image}')).shape) for image in images]
    if len(set(map(operator.itemgetter(-1), dim))) == 1:
      print(f'All elements in "{expression}" category has {dim[0][-1]} dimmensions')

  x = list(map(operator.itemgetter(0), dim))
  y = list(map(operator.itemgetter(1), dim))

  fig = plt.figure(figsize=(6, 2))
  plt.scatter(x,y)
  plt.title(set_name)
  plt.show()
In [21]:
display_dimmension_images_in_set(TRAIN_PATH, training_set, 'Training Set')
All elements in "happy" category has 3 dimmensions
All elements in "neutral" category has 3 dimmensions
All elements in "sad" category has 3 dimmensions
All elements in "surprise" category has 3 dimmensions
In [22]:
display_dimmension_images_in_set(TEST_PATH, testing_set, 'Testing Set')
All elements in "happy" category has 3 dimmensions
All elements in "neutral" category has 3 dimmensions
All elements in "sad" category has 3 dimmensions
All elements in "surprise" category has 3 dimmensions
In [23]:
display_dimmension_images_in_set(VAL_PATH, validation_set, 'Validation Set')
All elements in "happy" category has 3 dimmensions
All elements in "neutral" category has 3 dimmensions
All elements in "sad" category has 3 dimmensions
All elements in "surprise" category has 3 dimmensions

Checking Distribution of Classes¶

Think About It:

  • Are the classes equally distributed? If not, do you think the imbalance is too high? Will it be a problem as we progress?
  • Are there any Exploratory Data Analysis tasks that we can do here? Would they provide any meaningful insights?

⏩ Observations and Insights:

  • The testing dataset is nearly balanced distributed, the differences between the lowest and the highest categories are not significant. We can work with the current testing data.
  • The dataset in the validation set, seems not balanced, but this will not affect the training process, so no treatment technique will be applied.
  • If we need later to increase the number of images to avoid overfitting, we can use the ImageDataGenerator method as part of the Data Augmentation Technique.
  • It was also reviewed the dimmensions of the images. All of them has the same dimmensions: 48 X 48 pixels in 3 channels.

Creating our Data Loaders¶

In this section, we are creating data loaders that we will use as inputs to our Neural Network.

You have two options for the color_mode. You can set it to color_mode = 'rgb' or color_mode = 'grayscale'. You will need to try out both and see for yourself which one gives better performance.

In [24]:
def create_data_loaders(path, color_mode, batch_size=64):
  """Create the data loaders for training, testing and validation sets.
     It will be necessary to definte the color mode: rgb or grayscale.
  """
  # All images to be rescaled by 1/255.
  datagen_train = ImageDataGenerator(horizontal_flip = True,
                                     rescale=1./255,
                                     shear_range=0.3)
  datagen_test  = ImageDataGenerator(rescale = 1.0/255.)
  datagen_val  = ImageDataGenerator(rescale = 1.0/255.)

  # Generating the batches od the images
  default_params = {
      'target_size': (48, 48),
      'color_mode' : color_mode,
      'class_mode' : 'categorical',
      'classes'    : FACE_EXPRESSIONS,
      'shuffle'    : True
  }
  print(f'\nData with color_mode={color_mode}:')
  train_generator = datagen_train.flow_from_directory(f'{path}/train', batch_size=batch_size, **default_params)
  test_generator = datagen_train.flow_from_directory(f'{path}/test', batch_size=32, **default_params)
  val_generator = datagen_train.flow_from_directory(f'{path}/validation', batch_size=batch_size, **default_params)

  return train_generator, test_generator, val_generator
In [25]:
train_generator_rgb, _, _ = create_data_loaders(IMG_DIR, 'rgb')

# Taking a look at some examples of our augmented training data with color_mode='rgb'.
images, labels = next(train_generator_rgb)
fig, axes = plt.subplots(1, 8, figsize = (10, 2))
for (image, label, ax) in zip(images, labels, axes.flatten()):
  ax.imshow(image)
  ax.set_title(FACE_EXPRESSIONS[list(label).index(1)], fontsize=8)
  ax.axis('off')
plt.show()
Data with color_mode=rgb:
Found 15109 images belonging to 4 classes.
Found 127 images belonging to 4 classes.
Found 4977 images belonging to 4 classes.
In [26]:
train_generator_gray, _, _ = create_data_loaders(IMG_DIR, 'grayscale')

# Taking a look at some examples of our augmented training data with color_mode='grayscale'.
images, labels = next(train_generator_gray)
fig, axes = plt.subplots(1, 8, figsize = (10, 2))
for (image, label, ax) in zip(images, labels, axes.flatten()):
  ax.imshow(image)
  ax.set_title(FACE_EXPRESSIONS[list(label).index(1)], fontsize=8)
  ax.axis('off')
plt.show()

# Using proper cmap
fig, axes = plt.subplots(1, 8, figsize = (10, 2))
for (image, label, ax) in zip(images, labels, axes.flatten()):
  ax.imshow(image, cmap='gray')
  ax.set_title(FACE_EXPRESSIONS[list(label).index(1)], fontsize=8)
  ax.axis('off')
plt.show()
Data with color_mode=grayscale:
Found 15109 images belonging to 4 classes.
Found 127 images belonging to 4 classes.
Found 4977 images belonging to 4 classes.

Environment preparation¶

In [27]:
def save_object(obj, filename):
  """To save objects that can be loaded later"""
  with open(filename, 'wb') as f:
    pickle.dump(obj, f, pickle.HIGHEST_PROTOCOL)
In [28]:
def metrics_score(actual, predicted, model_name, history):
  """Calculating metrics to later compare performance of the model.
  It also prints the classification report and the confusion matrix
  """
  print(classification_report(actual, predicted, zero_division=0, target_names=FACE_EXPRESSIONS))
  cm = confusion_matrix(actual, predicted)

  plt.figure(figsize = (8, 5))
  sns.heatmap(cm, annot=True,  fmt='.0f', xticklabels=FACE_EXPRESSIONS, yticklabels=FACE_EXPRESSIONS)
  plt.ylabel('Actual')
  plt.xlabel('Predicted')
  plt.show()

  # Creating a dataframe of metrics
  index = np.argmax(history.history['val_accuracy'])
  df = pd.DataFrame(
      data={
          "Accuracy Train": history.history['accuracy'][index],
          "Accuracy Val": history.history['val_accuracy'][index],
          "Accuracy Test": accuracy_score(actual, predicted)
      },
      index=[model_name],
  )
  return df
In [29]:
def plot_model_result(result, model_name):
    """    Plot the accuracy and loss values of the model.
    """
    fig = plt.figure(figsize = (10, 3))

    # Plotting the Accuracy
    plt.subplot(1, 2, 1)
    plt.plot(result.history['accuracy'])
    plt.plot(result.history['val_accuracy'])
    plt.ylabel('Accuracy')
    plt.xlabel('Epoch')
    plt.legend(['train', 'validation'], loc='upper left')

    # Plotting the Loss
    plt.subplot(1, 2, 2)
    plt.plot(result.history['loss'])
    plt.plot(result.history['val_loss'])
    plt.ylabel('Loss')
    plt.xlabel('Epoch')
    plt.legend(['train', 'validation'], loc='upper left')

    fig.suptitle(model_name)
    plt.show()
In [30]:
def init_environment():
  # Clearing previous TF sessions
  backend.clear_session()

  # Fixing the seed for random number generators
  os.environ['PYTHONHASHSEED'] = str(2)
  np.random.seed(SEED)
  random.seed(SEED)
  tf.random.set_seed(SEED)
  print('Initializing TF Session and Seed')

⏩ Steps to follow in the model building:

  1. Initialize the environment
  2. Build the model
  3. Compile tne model
  4. Train the Model on the training set
  5. Evaluate the model on the testing set
  6. Tunning if necessary (Repeat steps 2-6)
In [31]:
def model_building_and_evaluating_process(model_builder, model_base_name, batch_size=64,
                                          include_grayscale=True, include_rgb=True, epochs=20, tune=0):
  """Execute the following steps: (1) Initialize the environment, (2) Build and compile the model based on the
  model builder method, (3) Train the model on the trainning set, and (4) Evaluate the model on the testing set
  """
  model_sets = []
  if include_rgb:
    model_sets.append(('RGB',  (48, 48, 3)))
  if include_grayscale:
    model_sets.append(('GRAY', (48, 48, 1)))

  model_metrics = pd.DataFrame({})
  model_history = {}
  fitted_model = {}

  for img_set, input_shape in model_sets:
    model_name = f'{model_base_name} - {img_set}'
    print('---------------------------------------------')
    print(model_name)
    print('---------------------------------------------')

    # Step 1: Initialize the environment
    init_environment()
    train_set, test_set, val_set = create_data_loaders(IMG_DIR, color_mode='rgb' if img_set=='RGB' else 'grayscale',
                                                       batch_size=batch_size)
    X_test, y_test = test_set.next()
    print('\n\n')

    # Step 2: Build and compile the model
    if tune:
      model = model_builder(input_shape=input_shape, tune=tune)
    else:
      model = model_builder(input_shape=input_shape)
    print('\n\n')

    # Step 3: Train the model on the training set
    checkpoint = ModelCheckpoint(f'{MODEL_DIR}/{model_name}.h5', monitor='val_accuracy', verbose=0, save_best_only=True, mode='max')
    early_stopping = EarlyStopping(monitor='val_loss', min_delta=0, patience=5, verbose=0, restore_best_weights=True)
    reduce_learningrate = ReduceLROnPlateau(monitor='val_loss', factor=0.2, patience=3, verbose=0, min_delta=0.0001)

    history = model.fit(train_set,
                        validation_data=val_set,
                        epochs=epochs,
                        callbacks=[early_stopping, checkpoint, reduce_learningrate])
    print('\n\n')

    # Step 4: Evaluate the model on the testing set
    model.evaluate(X_test, y_test, verbose=1)

    y_pred = np.argmax(model.predict(X_test), axis=1)
    y_test = np.argmax(y_test, axis=1)
    print('Actual    :', y_test)
    print('Prediction:', y_pred)
    print('\n\n')

    results = metrics_score(y_test, y_pred, model_name, history)
    model_metrics = pd.concat([model_metrics, results])
    print('\n\n')

    fitted_model[model_name] = model
    model_history[model_name] = history
    plot_model_result(history, model_name)
    print('\n\n')

  return fitted_model, model_metrics, model_history
In [32]:
def loss_and_accuracy_comparisson(model_history, data_type='Training and Validation Set'):
  """Visualizing the loss and the accuracy on the train and the
  validation data for all executed models.
  The "data_type" to plot could be "Training and Validation Set", "Validation Set", "Training Set".
  """

  fig, (ax1, ax2) = plt.subplots(1, 2, figsize = (10, 3))

  for model_name, history in model_history.items():
    if data_type in ["Training and Validation Set", "Training Set"]:
      ax1.plot(history.history['accuracy'], ls='-', lw=1, label=f'{model_name}')
    if data_type in ["Training and Validation Set", "Validation Set"]:
      ax1.plot(history.history['val_accuracy'], ls='--', lw=1, label=f'{model_name} val')
    ax1.set_ylabel('Accuracy', fontsize=8)
    ax1.set_xlabel('Epochs', fontsize=8)
    ax1.legend(fontsize=6)
    ax1.set_title('Accuracy Comparisson', fontsize=8)
    ax1.xaxis.set_tick_params(labelsize=7)
    ax1.yaxis.set_tick_params(labelsize=7)

    if data_type in ["Training and Validation Set", "Training Set"]:
      ax2.plot(history.history['loss'], ls='-', lw=1, label=f'{model_name}')
    if data_type in ["Training and Validation Set", "Validation Set"]:
      ax2.plot(history.history['val_loss'], ls='--', lw=1, label=f'{model_name} val')
    ax2.set_ylabel('Loss', fontsize=8)
    ax2.set_xlabel('Epochs', fontsize=8)
    ax2.legend(fontsize=6)
    ax2.set_title('Loss Comparisson', fontsize=8)
    ax2.xaxis.set_tick_params(labelsize=7)
    ax2.yaxis.set_tick_params(labelsize=7)

  fig.suptitle(f'Model Comparisson - {data_type}', fontsize=10)
  plt.tight_layout()
  plt.show()

Model Building¶

Think About It:

  • Are Convolutional Neural Networks the right approach? Should we have gone with Artificial Neural Networks instead?
  • What are the advantages of CNNs over ANNs and are they applicable here?

⏩ Insights:

  • For image classification problems, I consider that the CNN (Convolutional Neural Networks) approach is better than the ANN (Artificial Neural Networks) because:
    • CNNs helps us to capture easly the relevant features from an image, ignoring any spatial and translational transformations.
    • With the use of filters in a CNN, the model can reduce the dimensionality of the image and extract only the required information.
    • Also, CNN filters requires less trainable parameters in comparison to the ANN, this mean a computational advantage.

Model 1: Base Neural Network¶


In [33]:
# Model 1: Base Neural Network
def model_builder_base_cnn(input_shape):
  """Creating the Base Neural Network.
  """
  model = Sequential([
      Conv2D(filters=64, kernel_size=2, activation='relu', padding='same', input_shape=input_shape),
      MaxPooling2D(pool_size=2),
      Dropout(0.2),
      Conv2D(filters=32, kernel_size=2, activation='relu', padding='same'),
      MaxPooling2D(pool_size=2),
      Dropout(0.2),
      Conv2D(filters=32, kernel_size=2, activation='relu', padding='same'),
      MaxPooling2D(pool_size=2),
      Dropout(0.2),

      Flatten(),
      Dense(512, activation='relu'),
      Dropout(0.4),
      Dense(4, activation='softmax')
  ])

  # Compile model
  optimizer = Adam(learning_rate = 0.001)        # Using SGD Optimizer
  model.compile(optimizer=optimizer, loss='categorical_crossentropy', metrics=['accuracy'])

  # Generating the summary of the model
  print(model.summary())
  return model
In [34]:
batch_size = 32
model_base_cnn, metrics_base_cnn, history_base_cnn = model_building_and_evaluating_process(
    model_builder_base_cnn, f'Model 1: Base Neural Network ({batch_size})', batch_size=batch_size, epochs=20
)
---------------------------------------------
Model 1: Base Neural Network (32) - RGB
---------------------------------------------
Initializing TF Session and Seed

Data with color_mode=rgb:
Found 15109 images belonging to 4 classes.
Found 127 images belonging to 4 classes.
Found 4977 images belonging to 4 classes.



Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d (Conv2D)             (None, 48, 48, 64)        832       
                                                                 
 max_pooling2d (MaxPooling2D  (None, 24, 24, 64)       0         
 )                                                               
                                                                 
 dropout (Dropout)           (None, 24, 24, 64)        0         
                                                                 
 conv2d_1 (Conv2D)           (None, 24, 24, 32)        8224      
                                                                 
 max_pooling2d_1 (MaxPooling  (None, 12, 12, 32)       0         
 2D)                                                             
                                                                 
 dropout_1 (Dropout)         (None, 12, 12, 32)        0         
                                                                 
 conv2d_2 (Conv2D)           (None, 12, 12, 32)        4128      
                                                                 
 max_pooling2d_2 (MaxPooling  (None, 6, 6, 32)         0         
 2D)                                                             
                                                                 
 dropout_2 (Dropout)         (None, 6, 6, 32)          0         
                                                                 
 flatten (Flatten)           (None, 1152)              0         
                                                                 
 dense (Dense)               (None, 512)               590336    
                                                                 
 dropout_3 (Dropout)         (None, 512)               0         
                                                                 
 dense_1 (Dense)             (None, 4)                 2052      
                                                                 
=================================================================
Total params: 605,572
Trainable params: 605,572
Non-trainable params: 0
_________________________________________________________________
None



Epoch 1/20
473/473 [==============================] - 28s 56ms/step - loss: 1.3195 - accuracy: 0.3512 - val_loss: 1.1722 - val_accuracy: 0.5055 - lr: 0.0010
Epoch 2/20
473/473 [==============================] - 26s 55ms/step - loss: 1.1272 - accuracy: 0.5053 - val_loss: 1.0094 - val_accuracy: 0.5763 - lr: 0.0010
Epoch 3/20
473/473 [==============================] - 26s 55ms/step - loss: 1.0300 - accuracy: 0.5603 - val_loss: 0.9365 - val_accuracy: 0.6156 - lr: 0.0010
Epoch 4/20
473/473 [==============================] - 26s 55ms/step - loss: 0.9791 - accuracy: 0.5833 - val_loss: 0.8959 - val_accuracy: 0.6381 - lr: 0.0010
Epoch 5/20
473/473 [==============================] - 26s 54ms/step - loss: 0.9427 - accuracy: 0.6008 - val_loss: 0.8576 - val_accuracy: 0.6426 - lr: 0.0010
Epoch 6/20
473/473 [==============================] - 26s 55ms/step - loss: 0.9150 - accuracy: 0.6117 - val_loss: 0.8524 - val_accuracy: 0.6460 - lr: 0.0010
Epoch 7/20
473/473 [==============================] - 26s 54ms/step - loss: 0.8823 - accuracy: 0.6273 - val_loss: 0.8281 - val_accuracy: 0.6558 - lr: 0.0010
Epoch 8/20
473/473 [==============================] - 26s 54ms/step - loss: 0.8707 - accuracy: 0.6345 - val_loss: 0.8111 - val_accuracy: 0.6628 - lr: 0.0010
Epoch 9/20
473/473 [==============================] - 25s 53ms/step - loss: 0.8465 - accuracy: 0.6435 - val_loss: 0.9083 - val_accuracy: 0.6211 - lr: 0.0010
Epoch 10/20
473/473 [==============================] - 26s 54ms/step - loss: 0.8320 - accuracy: 0.6503 - val_loss: 0.7770 - val_accuracy: 0.6815 - lr: 0.0010
Epoch 11/20
473/473 [==============================] - 25s 54ms/step - loss: 0.8169 - accuracy: 0.6613 - val_loss: 0.7744 - val_accuracy: 0.6795 - lr: 0.0010
Epoch 12/20
473/473 [==============================] - 26s 54ms/step - loss: 0.7962 - accuracy: 0.6653 - val_loss: 0.7571 - val_accuracy: 0.6908 - lr: 0.0010
Epoch 13/20
473/473 [==============================] - 26s 54ms/step - loss: 0.7925 - accuracy: 0.6695 - val_loss: 0.7417 - val_accuracy: 0.6934 - lr: 0.0010
Epoch 14/20
473/473 [==============================] - 26s 54ms/step - loss: 0.7785 - accuracy: 0.6754 - val_loss: 0.7430 - val_accuracy: 0.6952 - lr: 0.0010
Epoch 15/20
473/473 [==============================] - 25s 53ms/step - loss: 0.7664 - accuracy: 0.6851 - val_loss: 0.7474 - val_accuracy: 0.6946 - lr: 0.0010
Epoch 16/20
473/473 [==============================] - 26s 54ms/step - loss: 0.7591 - accuracy: 0.6842 - val_loss: 0.7310 - val_accuracy: 0.7103 - lr: 0.0010
Epoch 17/20
473/473 [==============================] - 25s 53ms/step - loss: 0.7407 - accuracy: 0.6998 - val_loss: 0.7458 - val_accuracy: 0.6922 - lr: 0.0010
Epoch 18/20
473/473 [==============================] - 25s 54ms/step - loss: 0.7391 - accuracy: 0.6977 - val_loss: 0.7134 - val_accuracy: 0.7153 - lr: 0.0010
Epoch 19/20
473/473 [==============================] - 25s 53ms/step - loss: 0.7329 - accuracy: 0.6989 - val_loss: 0.7137 - val_accuracy: 0.7137 - lr: 0.0010
Epoch 20/20
473/473 [==============================] - 26s 54ms/step - loss: 0.7171 - accuracy: 0.7059 - val_loss: 0.7141 - val_accuracy: 0.7191 - lr: 0.0010



1/1 [==============================] - 0s 33ms/step - loss: 0.6004 - accuracy: 0.7812
1/1 [==============================] - 0s 111ms/step
Actual    : [0 1 3 0 1 3 2 1 0 2 0 3 2 2 3 0 3 2 0 2 1 3 2 2 0 0 3 0 1 2 0 2]
Prediction: [0 1 3 0 2 3 2 1 0 1 1 3 2 2 3 0 3 2 0 2 2 3 2 2 0 2 1 0 1 1 0 2]



              precision    recall  f1-score   support

       happy       1.00      0.80      0.89        10
     neutral       0.43      0.60      0.50         5
         sad       0.73      0.80      0.76        10
    surprise       1.00      0.86      0.92         7

    accuracy                           0.78        32
   macro avg       0.79      0.76      0.77        32
weighted avg       0.83      0.78      0.80        32





---------------------------------------------
Model 1: Base Neural Network (32) - GRAY
---------------------------------------------
Initializing TF Session and Seed

Data with color_mode=grayscale:
Found 15109 images belonging to 4 classes.
Found 127 images belonging to 4 classes.
Found 4977 images belonging to 4 classes.



Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d (Conv2D)             (None, 48, 48, 64)        320       
                                                                 
 max_pooling2d (MaxPooling2D  (None, 24, 24, 64)       0         
 )                                                               
                                                                 
 dropout (Dropout)           (None, 24, 24, 64)        0         
                                                                 
 conv2d_1 (Conv2D)           (None, 24, 24, 32)        8224      
                                                                 
 max_pooling2d_1 (MaxPooling  (None, 12, 12, 32)       0         
 2D)                                                             
                                                                 
 dropout_1 (Dropout)         (None, 12, 12, 32)        0         
                                                                 
 conv2d_2 (Conv2D)           (None, 12, 12, 32)        4128      
                                                                 
 max_pooling2d_2 (MaxPooling  (None, 6, 6, 32)         0         
 2D)                                                             
                                                                 
 dropout_2 (Dropout)         (None, 6, 6, 32)          0         
                                                                 
 flatten (Flatten)           (None, 1152)              0         
                                                                 
 dense (Dense)               (None, 512)               590336    
                                                                 
 dropout_3 (Dropout)         (None, 512)               0         
                                                                 
 dense_1 (Dense)             (None, 4)                 2052      
                                                                 
=================================================================
Total params: 605,060
Trainable params: 605,060
Non-trainable params: 0
_________________________________________________________________
None



Epoch 1/20
473/473 [==============================] - 18s 36ms/step - loss: 1.2927 - accuracy: 0.3759 - val_loss: 1.1302 - val_accuracy: 0.5214 - lr: 0.0010
Epoch 2/20
473/473 [==============================] - 17s 35ms/step - loss: 1.1190 - accuracy: 0.5128 - val_loss: 1.0353 - val_accuracy: 0.5712 - lr: 0.0010
Epoch 3/20
473/473 [==============================] - 17s 35ms/step - loss: 1.0362 - accuracy: 0.5545 - val_loss: 0.9599 - val_accuracy: 0.5931 - lr: 0.0010
Epoch 4/20
473/473 [==============================] - 17s 35ms/step - loss: 0.9930 - accuracy: 0.5734 - val_loss: 0.9279 - val_accuracy: 0.6078 - lr: 0.0010
Epoch 5/20
473/473 [==============================] - 17s 36ms/step - loss: 0.9548 - accuracy: 0.5902 - val_loss: 0.8821 - val_accuracy: 0.6331 - lr: 0.0010
Epoch 6/20
473/473 [==============================] - 17s 35ms/step - loss: 0.9330 - accuracy: 0.6047 - val_loss: 0.8614 - val_accuracy: 0.6458 - lr: 0.0010
Epoch 7/20
473/473 [==============================] - 17s 35ms/step - loss: 0.9047 - accuracy: 0.6176 - val_loss: 0.8477 - val_accuracy: 0.6552 - lr: 0.0010
Epoch 8/20
473/473 [==============================] - 17s 36ms/step - loss: 0.8879 - accuracy: 0.6253 - val_loss: 0.8345 - val_accuracy: 0.6566 - lr: 0.0010
Epoch 9/20
473/473 [==============================] - 16s 35ms/step - loss: 0.8648 - accuracy: 0.6349 - val_loss: 0.8628 - val_accuracy: 0.6510 - lr: 0.0010
Epoch 10/20
473/473 [==============================] - 17s 35ms/step - loss: 0.8520 - accuracy: 0.6444 - val_loss: 0.7941 - val_accuracy: 0.6789 - lr: 0.0010
Epoch 11/20
473/473 [==============================] - 16s 35ms/step - loss: 0.8362 - accuracy: 0.6494 - val_loss: 0.7748 - val_accuracy: 0.6781 - lr: 0.0010
Epoch 12/20
473/473 [==============================] - 17s 35ms/step - loss: 0.8236 - accuracy: 0.6538 - val_loss: 0.7743 - val_accuracy: 0.6805 - lr: 0.0010
Epoch 13/20
473/473 [==============================] - 17s 35ms/step - loss: 0.8113 - accuracy: 0.6628 - val_loss: 0.7691 - val_accuracy: 0.6827 - lr: 0.0010
Epoch 14/20
473/473 [==============================] - 16s 34ms/step - loss: 0.7949 - accuracy: 0.6744 - val_loss: 0.7791 - val_accuracy: 0.6783 - lr: 0.0010
Epoch 15/20
473/473 [==============================] - 16s 34ms/step - loss: 0.7895 - accuracy: 0.6734 - val_loss: 0.7701 - val_accuracy: 0.6821 - lr: 0.0010
Epoch 16/20
473/473 [==============================] - 16s 35ms/step - loss: 0.7774 - accuracy: 0.6793 - val_loss: 0.7605 - val_accuracy: 0.6960 - lr: 0.0010
Epoch 17/20
473/473 [==============================] - 16s 34ms/step - loss: 0.7618 - accuracy: 0.6881 - val_loss: 0.7565 - val_accuracy: 0.6890 - lr: 0.0010
Epoch 18/20
473/473 [==============================] - 16s 35ms/step - loss: 0.7623 - accuracy: 0.6848 - val_loss: 0.7480 - val_accuracy: 0.7022 - lr: 0.0010
Epoch 19/20
473/473 [==============================] - 16s 34ms/step - loss: 0.7532 - accuracy: 0.6906 - val_loss: 0.7451 - val_accuracy: 0.7010 - lr: 0.0010
Epoch 20/20
473/473 [==============================] - 16s 34ms/step - loss: 0.7371 - accuracy: 0.6977 - val_loss: 0.7562 - val_accuracy: 0.6924 - lr: 0.0010



1/1 [==============================] - 0s 27ms/step - loss: 0.7368 - accuracy: 0.7812
1/1 [==============================] - 0s 81ms/step
Actual    : [0 1 3 0 1 3 2 1 0 2 0 3 2 2 3 0 3 2 0 2 1 3 2 2 0 0 3 0 1 2 0 2]
Prediction: [0 1 3 0 1 3 2 1 0 2 1 3 2 2 3 0 3 2 0 2 2 2 1 2 0 1 1 0 1 1 0 2]



              precision    recall  f1-score   support

       happy       1.00      0.80      0.89        10
     neutral       0.44      0.80      0.57         5
         sad       0.80      0.80      0.80        10
    surprise       1.00      0.71      0.83         7

    accuracy                           0.78        32
   macro avg       0.81      0.78      0.77        32
weighted avg       0.85      0.78      0.80        32





In [35]:
batch_size = 64
model, results, history = model_building_and_evaluating_process(model_builder_base_cnn, f'Model 1: Base Neural Network ({batch_size})',
                                                                batch_size=batch_size, epochs=20)
model_base_cnn.update(model)
metrics_base_cnn = pd.concat([metrics_base_cnn, results])
history_base_cnn.update(history)
---------------------------------------------
Model 1: Base Neural Network (64) - RGB
---------------------------------------------
Initializing TF Session and Seed

Data with color_mode=rgb:
Found 15109 images belonging to 4 classes.
Found 127 images belonging to 4 classes.
Found 4977 images belonging to 4 classes.



Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d (Conv2D)             (None, 48, 48, 64)        832       
                                                                 
 max_pooling2d (MaxPooling2D  (None, 24, 24, 64)       0         
 )                                                               
                                                                 
 dropout (Dropout)           (None, 24, 24, 64)        0         
                                                                 
 conv2d_1 (Conv2D)           (None, 24, 24, 32)        8224      
                                                                 
 max_pooling2d_1 (MaxPooling  (None, 12, 12, 32)       0         
 2D)                                                             
                                                                 
 dropout_1 (Dropout)         (None, 12, 12, 32)        0         
                                                                 
 conv2d_2 (Conv2D)           (None, 12, 12, 32)        4128      
                                                                 
 max_pooling2d_2 (MaxPooling  (None, 6, 6, 32)         0         
 2D)                                                             
                                                                 
 dropout_2 (Dropout)         (None, 6, 6, 32)          0         
                                                                 
 flatten (Flatten)           (None, 1152)              0         
                                                                 
 dense (Dense)               (None, 512)               590336    
                                                                 
 dropout_3 (Dropout)         (None, 512)               0         
                                                                 
 dense_1 (Dense)             (None, 4)                 2052      
                                                                 
=================================================================
Total params: 605,572
Trainable params: 605,572
Non-trainable params: 0
_________________________________________________________________
None



Epoch 1/20
237/237 [==============================] - 26s 105ms/step - loss: 1.3350 - accuracy: 0.3336 - val_loss: 1.2285 - val_accuracy: 0.4406 - lr: 0.0010
Epoch 2/20
237/237 [==============================] - 25s 104ms/step - loss: 1.1514 - accuracy: 0.4954 - val_loss: 1.0440 - val_accuracy: 0.5630 - lr: 0.0010
Epoch 3/20
237/237 [==============================] - 25s 104ms/step - loss: 1.0589 - accuracy: 0.5458 - val_loss: 0.9748 - val_accuracy: 0.5929 - lr: 0.0010
Epoch 4/20
237/237 [==============================] - 25s 104ms/step - loss: 1.0048 - accuracy: 0.5693 - val_loss: 0.9210 - val_accuracy: 0.6174 - lr: 0.0010
Epoch 5/20
237/237 [==============================] - 25s 104ms/step - loss: 0.9633 - accuracy: 0.5898 - val_loss: 0.8824 - val_accuracy: 0.6305 - lr: 0.0010
Epoch 6/20
237/237 [==============================] - 25s 105ms/step - loss: 0.9483 - accuracy: 0.5953 - val_loss: 0.8572 - val_accuracy: 0.6438 - lr: 0.0010
Epoch 7/20
237/237 [==============================] - 25s 104ms/step - loss: 0.9134 - accuracy: 0.6138 - val_loss: 0.8352 - val_accuracy: 0.6582 - lr: 0.0010
Epoch 8/20
237/237 [==============================] - 24s 102ms/step - loss: 0.8938 - accuracy: 0.6241 - val_loss: 0.8455 - val_accuracy: 0.6524 - lr: 0.0010
Epoch 9/20
237/237 [==============================] - 25s 104ms/step - loss: 0.8755 - accuracy: 0.6327 - val_loss: 0.8110 - val_accuracy: 0.6707 - lr: 0.0010
Epoch 10/20
237/237 [==============================] - 24s 103ms/step - loss: 0.8609 - accuracy: 0.6434 - val_loss: 0.8108 - val_accuracy: 0.6631 - lr: 0.0010
Epoch 11/20
237/237 [==============================] - 24s 103ms/step - loss: 0.8399 - accuracy: 0.6482 - val_loss: 0.8028 - val_accuracy: 0.6705 - lr: 0.0010
Epoch 12/20
237/237 [==============================] - 25s 104ms/step - loss: 0.8305 - accuracy: 0.6559 - val_loss: 0.7610 - val_accuracy: 0.6892 - lr: 0.0010
Epoch 13/20
237/237 [==============================] - 24s 102ms/step - loss: 0.8115 - accuracy: 0.6662 - val_loss: 0.7873 - val_accuracy: 0.6729 - lr: 0.0010
Epoch 14/20
237/237 [==============================] - 25s 104ms/step - loss: 0.8060 - accuracy: 0.6654 - val_loss: 0.7576 - val_accuracy: 0.6902 - lr: 0.0010
Epoch 15/20
237/237 [==============================] - 25s 104ms/step - loss: 0.7871 - accuracy: 0.6735 - val_loss: 0.7419 - val_accuracy: 0.6998 - lr: 0.0010
Epoch 16/20
237/237 [==============================] - 24s 103ms/step - loss: 0.7744 - accuracy: 0.6811 - val_loss: 0.7882 - val_accuracy: 0.6783 - lr: 0.0010
Epoch 17/20
237/237 [==============================] - 24s 102ms/step - loss: 0.7680 - accuracy: 0.6866 - val_loss: 0.7927 - val_accuracy: 0.6757 - lr: 0.0010
Epoch 18/20
237/237 [==============================] - 24s 102ms/step - loss: 0.7544 - accuracy: 0.6879 - val_loss: 0.7602 - val_accuracy: 0.6854 - lr: 0.0010
Epoch 19/20
237/237 [==============================] - 25s 104ms/step - loss: 0.7134 - accuracy: 0.7036 - val_loss: 0.7214 - val_accuracy: 0.7054 - lr: 2.0000e-04
Epoch 20/20
237/237 [==============================] - 25s 103ms/step - loss: 0.7070 - accuracy: 0.7113 - val_loss: 0.7120 - val_accuracy: 0.7133 - lr: 2.0000e-04



1/1 [==============================] - 0s 30ms/step - loss: 0.6727 - accuracy: 0.7188
1/1 [==============================] - 0s 391ms/step
Actual    : [0 1 3 0 1 3 2 1 0 2 0 3 2 2 3 0 3 2 0 2 1 3 2 2 0 0 3 0 1 2 0 2]
Prediction: [0 1 3 0 2 3 2 1 0 1 2 3 2 0 3 0 3 2 0 2 2 3 1 2 0 1 1 0 1 1 0 2]



              precision    recall  f1-score   support

       happy       0.89      0.80      0.84        10
     neutral       0.38      0.60      0.46         5
         sad       0.67      0.60      0.63        10
    surprise       1.00      0.86      0.92         7

    accuracy                           0.72        32
   macro avg       0.73      0.71      0.71        32
weighted avg       0.76      0.72      0.73        32





---------------------------------------------
Model 1: Base Neural Network (64) - GRAY
---------------------------------------------
Initializing TF Session and Seed

Data with color_mode=grayscale:
Found 15109 images belonging to 4 classes.
Found 127 images belonging to 4 classes.
Found 4977 images belonging to 4 classes.



Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d (Conv2D)             (None, 48, 48, 64)        320       
                                                                 
 max_pooling2d (MaxPooling2D  (None, 24, 24, 64)       0         
 )                                                               
                                                                 
 dropout (Dropout)           (None, 24, 24, 64)        0         
                                                                 
 conv2d_1 (Conv2D)           (None, 24, 24, 32)        8224      
                                                                 
 max_pooling2d_1 (MaxPooling  (None, 12, 12, 32)       0         
 2D)                                                             
                                                                 
 dropout_1 (Dropout)         (None, 12, 12, 32)        0         
                                                                 
 conv2d_2 (Conv2D)           (None, 12, 12, 32)        4128      
                                                                 
 max_pooling2d_2 (MaxPooling  (None, 6, 6, 32)         0         
 2D)                                                             
                                                                 
 dropout_2 (Dropout)         (None, 6, 6, 32)          0         
                                                                 
 flatten (Flatten)           (None, 1152)              0         
                                                                 
 dense (Dense)               (None, 512)               590336    
                                                                 
 dropout_3 (Dropout)         (None, 512)               0         
                                                                 
 dense_1 (Dense)             (None, 4)                 2052      
                                                                 
=================================================================
Total params: 605,060
Trainable params: 605,060
Non-trainable params: 0
_________________________________________________________________
None



Epoch 1/20
237/237 [==============================] - 17s 68ms/step - loss: 1.3277 - accuracy: 0.3409 - val_loss: 1.1836 - val_accuracy: 0.4762 - lr: 0.0010
Epoch 2/20
237/237 [==============================] - 16s 67ms/step - loss: 1.1535 - accuracy: 0.4954 - val_loss: 1.0556 - val_accuracy: 0.5644 - lr: 0.0010
Epoch 3/20
237/237 [==============================] - 16s 67ms/step - loss: 1.0683 - accuracy: 0.5415 - val_loss: 0.9965 - val_accuracy: 0.5857 - lr: 0.0010
Epoch 4/20
237/237 [==============================] - 16s 67ms/step - loss: 1.0169 - accuracy: 0.5640 - val_loss: 0.9461 - val_accuracy: 0.6102 - lr: 0.0010
Epoch 5/20
237/237 [==============================] - 16s 67ms/step - loss: 0.9744 - accuracy: 0.5848 - val_loss: 0.9253 - val_accuracy: 0.6225 - lr: 0.0010
Epoch 6/20
237/237 [==============================] - 16s 67ms/step - loss: 0.9466 - accuracy: 0.5993 - val_loss: 0.8794 - val_accuracy: 0.6303 - lr: 0.0010
Epoch 7/20
237/237 [==============================] - 16s 67ms/step - loss: 0.9149 - accuracy: 0.6150 - val_loss: 0.8528 - val_accuracy: 0.6522 - lr: 0.0010
Epoch 8/20
237/237 [==============================] - 16s 66ms/step - loss: 0.8908 - accuracy: 0.6288 - val_loss: 0.8313 - val_accuracy: 0.6532 - lr: 0.0010
Epoch 9/20
237/237 [==============================] - 16s 67ms/step - loss: 0.8729 - accuracy: 0.6351 - val_loss: 0.8082 - val_accuracy: 0.6683 - lr: 0.0010
Epoch 10/20
237/237 [==============================] - 16s 65ms/step - loss: 0.8685 - accuracy: 0.6331 - val_loss: 0.8210 - val_accuracy: 0.6546 - lr: 0.0010
Epoch 11/20
237/237 [==============================] - 16s 65ms/step - loss: 0.8466 - accuracy: 0.6473 - val_loss: 0.8030 - val_accuracy: 0.6631 - lr: 0.0010
Epoch 12/20
237/237 [==============================] - 16s 68ms/step - loss: 0.8272 - accuracy: 0.6552 - val_loss: 0.7729 - val_accuracy: 0.6868 - lr: 0.0010
Epoch 13/20
237/237 [==============================] - 16s 65ms/step - loss: 0.8248 - accuracy: 0.6583 - val_loss: 0.8256 - val_accuracy: 0.6578 - lr: 0.0010
Epoch 14/20
237/237 [==============================] - 16s 67ms/step - loss: 0.8069 - accuracy: 0.6639 - val_loss: 0.7737 - val_accuracy: 0.6878 - lr: 0.0010
Epoch 15/20
237/237 [==============================] - 16s 67ms/step - loss: 0.7949 - accuracy: 0.6746 - val_loss: 0.7562 - val_accuracy: 0.6928 - lr: 0.0010
Epoch 16/20
237/237 [==============================] - 15s 65ms/step - loss: 0.7805 - accuracy: 0.6791 - val_loss: 0.7726 - val_accuracy: 0.6809 - lr: 0.0010
Epoch 17/20
237/237 [==============================] - 15s 65ms/step - loss: 0.7762 - accuracy: 0.6820 - val_loss: 0.7845 - val_accuracy: 0.6767 - lr: 0.0010
Epoch 18/20
237/237 [==============================] - 16s 65ms/step - loss: 0.7628 - accuracy: 0.6854 - val_loss: 0.7865 - val_accuracy: 0.6735 - lr: 0.0010
Epoch 19/20
237/237 [==============================] - 16s 67ms/step - loss: 0.7250 - accuracy: 0.7030 - val_loss: 0.7362 - val_accuracy: 0.7018 - lr: 2.0000e-04
Epoch 20/20
237/237 [==============================] - 16s 67ms/step - loss: 0.7234 - accuracy: 0.7088 - val_loss: 0.7283 - val_accuracy: 0.7038 - lr: 2.0000e-04



1/1 [==============================] - 0s 28ms/step - loss: 0.7231 - accuracy: 0.6250
1/1 [==============================] - 0s 84ms/step
Actual    : [0 1 3 0 1 3 2 1 0 2 0 3 2 2 3 0 3 2 0 2 1 3 2 2 0 0 3 0 1 2 0 2]
Prediction: [0 1 0 0 2 3 2 1 0 1 2 3 2 0 3 0 3 2 0 2 2 2 1 1 0 2 1 0 1 1 0 2]



              precision    recall  f1-score   support

       happy       0.80      0.80      0.80        10
     neutral       0.38      0.60      0.46         5
         sad       0.50      0.50      0.50        10
    surprise       1.00      0.57      0.73         7

    accuracy                           0.62        32
   macro avg       0.67      0.62      0.62        32
weighted avg       0.68      0.62      0.64        32





In [36]:
batch_size = 128
model, results, history = model_building_and_evaluating_process(model_builder_base_cnn, f'Model 1: Base Neural Network ({batch_size})',
                                                                batch_size=batch_size, epochs=20)
model_base_cnn.update(model)
metrics_base_cnn = pd.concat([metrics_base_cnn, results])
history_base_cnn.update(history)
---------------------------------------------
Model 1: Base Neural Network (128) - RGB
---------------------------------------------
Initializing TF Session and Seed

Data with color_mode=rgb:
Found 15109 images belonging to 4 classes.
Found 127 images belonging to 4 classes.
Found 4977 images belonging to 4 classes.



Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d (Conv2D)             (None, 48, 48, 64)        832       
                                                                 
 max_pooling2d (MaxPooling2D  (None, 24, 24, 64)       0         
 )                                                               
                                                                 
 dropout (Dropout)           (None, 24, 24, 64)        0         
                                                                 
 conv2d_1 (Conv2D)           (None, 24, 24, 32)        8224      
                                                                 
 max_pooling2d_1 (MaxPooling  (None, 12, 12, 32)       0         
 2D)                                                             
                                                                 
 dropout_1 (Dropout)         (None, 12, 12, 32)        0         
                                                                 
 conv2d_2 (Conv2D)           (None, 12, 12, 32)        4128      
                                                                 
 max_pooling2d_2 (MaxPooling  (None, 6, 6, 32)         0         
 2D)                                                             
                                                                 
 dropout_2 (Dropout)         (None, 6, 6, 32)          0         
                                                                 
 flatten (Flatten)           (None, 1152)              0         
                                                                 
 dense (Dense)               (None, 512)               590336    
                                                                 
 dropout_3 (Dropout)         (None, 512)               0         
                                                                 
 dense_1 (Dense)             (None, 4)                 2052      
                                                                 
=================================================================
Total params: 605,572
Trainable params: 605,572
Non-trainable params: 0
_________________________________________________________________
None



Epoch 1/20
119/119 [==============================] - 26s 207ms/step - loss: 1.3598 - accuracy: 0.3080 - val_loss: 1.3520 - val_accuracy: 0.2773 - lr: 0.0010
Epoch 2/20
119/119 [==============================] - 24s 204ms/step - loss: 1.2317 - accuracy: 0.4331 - val_loss: 1.1257 - val_accuracy: 0.5280 - lr: 0.0010
Epoch 3/20
119/119 [==============================] - 24s 204ms/step - loss: 1.1147 - accuracy: 0.5164 - val_loss: 1.0227 - val_accuracy: 0.5728 - lr: 0.0010
Epoch 4/20
119/119 [==============================] - 24s 202ms/step - loss: 1.0479 - accuracy: 0.5464 - val_loss: 0.9704 - val_accuracy: 0.6014 - lr: 0.0010
Epoch 5/20
119/119 [==============================] - 24s 203ms/step - loss: 1.0082 - accuracy: 0.5658 - val_loss: 0.9355 - val_accuracy: 0.6180 - lr: 0.0010
Epoch 6/20
119/119 [==============================] - 24s 205ms/step - loss: 0.9609 - accuracy: 0.5912 - val_loss: 0.9181 - val_accuracy: 0.6217 - lr: 0.0010
Epoch 7/20
119/119 [==============================] - 24s 205ms/step - loss: 0.9404 - accuracy: 0.6039 - val_loss: 0.8763 - val_accuracy: 0.6371 - lr: 0.0010
Epoch 8/20
119/119 [==============================] - 24s 203ms/step - loss: 0.9191 - accuracy: 0.6122 - val_loss: 0.8453 - val_accuracy: 0.6560 - lr: 0.0010
Epoch 9/20
119/119 [==============================] - 24s 200ms/step - loss: 0.9035 - accuracy: 0.6227 - val_loss: 0.8501 - val_accuracy: 0.6540 - lr: 0.0010
Epoch 10/20
119/119 [==============================] - 24s 200ms/step - loss: 0.8799 - accuracy: 0.6290 - val_loss: 0.8724 - val_accuracy: 0.6391 - lr: 0.0010
Epoch 11/20
119/119 [==============================] - 24s 203ms/step - loss: 0.8736 - accuracy: 0.6312 - val_loss: 0.8216 - val_accuracy: 0.6739 - lr: 0.0010
Epoch 12/20
119/119 [==============================] - 24s 204ms/step - loss: 0.8619 - accuracy: 0.6380 - val_loss: 0.8065 - val_accuracy: 0.6805 - lr: 0.0010
Epoch 13/20
119/119 [==============================] - 24s 200ms/step - loss: 0.8513 - accuracy: 0.6451 - val_loss: 0.7790 - val_accuracy: 0.6805 - lr: 0.0010
Epoch 14/20
119/119 [==============================] - 24s 203ms/step - loss: 0.8334 - accuracy: 0.6513 - val_loss: 0.7787 - val_accuracy: 0.6835 - lr: 0.0010
Epoch 15/20
119/119 [==============================] - 24s 205ms/step - loss: 0.8248 - accuracy: 0.6558 - val_loss: 0.7671 - val_accuracy: 0.6928 - lr: 0.0010
Epoch 16/20
119/119 [==============================] - 24s 200ms/step - loss: 0.8166 - accuracy: 0.6628 - val_loss: 0.7877 - val_accuracy: 0.6890 - lr: 0.0010
Epoch 17/20
119/119 [==============================] - 24s 202ms/step - loss: 0.8073 - accuracy: 0.6650 - val_loss: 0.7626 - val_accuracy: 0.6952 - lr: 0.0010
Epoch 18/20
119/119 [==============================] - 24s 201ms/step - loss: 0.7965 - accuracy: 0.6692 - val_loss: 0.7757 - val_accuracy: 0.6870 - lr: 0.0010
Epoch 19/20
119/119 [==============================] - 24s 204ms/step - loss: 0.7800 - accuracy: 0.6832 - val_loss: 0.7544 - val_accuracy: 0.7006 - lr: 0.0010
Epoch 20/20
119/119 [==============================] - 24s 206ms/step - loss: 0.7760 - accuracy: 0.6832 - val_loss: 0.7409 - val_accuracy: 0.7018 - lr: 0.0010



1/1 [==============================] - 0s 27ms/step - loss: 0.6682 - accuracy: 0.7500
WARNING:tensorflow:5 out of the last 5 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7fcaa01143a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
1/1 [==============================] - 0s 86ms/step
Actual    : [0 1 3 0 1 3 2 1 0 2 0 3 2 2 3 0 3 2 0 2 1 3 2 2 0 0 3 0 1 2 0 2]
Prediction: [0 1 3 0 0 3 2 1 0 1 2 3 2 2 3 0 3 2 0 2 2 3 1 2 0 2 1 0 1 1 0 2]



              precision    recall  f1-score   support

       happy       0.89      0.80      0.84        10
     neutral       0.43      0.60      0.50         5
         sad       0.70      0.70      0.70        10
    surprise       1.00      0.86      0.92         7

    accuracy                           0.75        32
   macro avg       0.75      0.74      0.74        32
weighted avg       0.78      0.75      0.76        32





---------------------------------------------
Model 1: Base Neural Network (128) - GRAY
---------------------------------------------
Initializing TF Session and Seed

Data with color_mode=grayscale:
Found 15109 images belonging to 4 classes.
Found 127 images belonging to 4 classes.
Found 4977 images belonging to 4 classes.



Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d (Conv2D)             (None, 48, 48, 64)        320       
                                                                 
 max_pooling2d (MaxPooling2D  (None, 24, 24, 64)       0         
 )                                                               
                                                                 
 dropout (Dropout)           (None, 24, 24, 64)        0         
                                                                 
 conv2d_1 (Conv2D)           (None, 24, 24, 32)        8224      
                                                                 
 max_pooling2d_1 (MaxPooling  (None, 12, 12, 32)       0         
 2D)                                                             
                                                                 
 dropout_1 (Dropout)         (None, 12, 12, 32)        0         
                                                                 
 conv2d_2 (Conv2D)           (None, 12, 12, 32)        4128      
                                                                 
 max_pooling2d_2 (MaxPooling  (None, 6, 6, 32)         0         
 2D)                                                             
                                                                 
 dropout_2 (Dropout)         (None, 6, 6, 32)          0         
                                                                 
 flatten (Flatten)           (None, 1152)              0         
                                                                 
 dense (Dense)               (None, 512)               590336    
                                                                 
 dropout_3 (Dropout)         (None, 512)               0         
                                                                 
 dense_1 (Dense)             (None, 4)                 2052      
                                                                 
=================================================================
Total params: 605,060
Trainable params: 605,060
Non-trainable params: 0
_________________________________________________________________
None



Epoch 1/20
119/119 [==============================] - 17s 134ms/step - loss: 1.3531 - accuracy: 0.3193 - val_loss: 1.3037 - val_accuracy: 0.3824 - lr: 0.0010
Epoch 2/20
119/119 [==============================] - 16s 132ms/step - loss: 1.2397 - accuracy: 0.4326 - val_loss: 1.1524 - val_accuracy: 0.5101 - lr: 0.0010
Epoch 3/20
119/119 [==============================] - 16s 132ms/step - loss: 1.1440 - accuracy: 0.5014 - val_loss: 1.0613 - val_accuracy: 0.5634 - lr: 0.0010
Epoch 4/20
119/119 [==============================] - 16s 131ms/step - loss: 1.0848 - accuracy: 0.5296 - val_loss: 1.0169 - val_accuracy: 0.5813 - lr: 0.0010
Epoch 5/20
119/119 [==============================] - 16s 131ms/step - loss: 1.0316 - accuracy: 0.5626 - val_loss: 0.9673 - val_accuracy: 0.5998 - lr: 0.0010
Epoch 6/20
119/119 [==============================] - 16s 132ms/step - loss: 0.9936 - accuracy: 0.5797 - val_loss: 0.9314 - val_accuracy: 0.6148 - lr: 0.0010
Epoch 7/20
119/119 [==============================] - 16s 132ms/step - loss: 0.9713 - accuracy: 0.5880 - val_loss: 0.9358 - val_accuracy: 0.6170 - lr: 0.0010
Epoch 8/20
119/119 [==============================] - 16s 131ms/step - loss: 0.9474 - accuracy: 0.6008 - val_loss: 0.8824 - val_accuracy: 0.6432 - lr: 0.0010
Epoch 9/20
119/119 [==============================] - 16s 131ms/step - loss: 0.9232 - accuracy: 0.6106 - val_loss: 0.8665 - val_accuracy: 0.6462 - lr: 0.0010
Epoch 10/20
119/119 [==============================] - 15s 129ms/step - loss: 0.9045 - accuracy: 0.6182 - val_loss: 0.9163 - val_accuracy: 0.6074 - lr: 0.0010
Epoch 11/20
119/119 [==============================] - 16s 131ms/step - loss: 0.9012 - accuracy: 0.6183 - val_loss: 0.8631 - val_accuracy: 0.6476 - lr: 0.0010
Epoch 12/20
119/119 [==============================] - 16s 131ms/step - loss: 0.8793 - accuracy: 0.6354 - val_loss: 0.8176 - val_accuracy: 0.6679 - lr: 0.0010
Epoch 13/20
119/119 [==============================] - 15s 128ms/step - loss: 0.8718 - accuracy: 0.6347 - val_loss: 0.8207 - val_accuracy: 0.6635 - lr: 0.0010
Epoch 14/20
119/119 [==============================] - 15s 128ms/step - loss: 0.8559 - accuracy: 0.6441 - val_loss: 0.8093 - val_accuracy: 0.6667 - lr: 0.0010
Epoch 15/20
119/119 [==============================] - 16s 131ms/step - loss: 0.8443 - accuracy: 0.6516 - val_loss: 0.8021 - val_accuracy: 0.6733 - lr: 0.0010
Epoch 16/20
119/119 [==============================] - 15s 128ms/step - loss: 0.8346 - accuracy: 0.6558 - val_loss: 0.8442 - val_accuracy: 0.6586 - lr: 0.0010
Epoch 17/20
119/119 [==============================] - 15s 127ms/step - loss: 0.8406 - accuracy: 0.6487 - val_loss: 0.8018 - val_accuracy: 0.6679 - lr: 0.0010
Epoch 18/20
119/119 [==============================] - 16s 130ms/step - loss: 0.8326 - accuracy: 0.6560 - val_loss: 0.7731 - val_accuracy: 0.6866 - lr: 0.0010
Epoch 19/20
119/119 [==============================] - 15s 128ms/step - loss: 0.8094 - accuracy: 0.6638 - val_loss: 0.7811 - val_accuracy: 0.6839 - lr: 0.0010
Epoch 20/20
119/119 [==============================] - 16s 130ms/step - loss: 0.8025 - accuracy: 0.6681 - val_loss: 0.7662 - val_accuracy: 0.6868 - lr: 0.0010



1/1 [==============================] - 0s 28ms/step - loss: 0.7159 - accuracy: 0.6875
WARNING:tensorflow:6 out of the last 6 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7fca8027acb0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
1/1 [==============================] - 0s 87ms/step
Actual    : [0 1 3 0 1 3 2 1 0 2 0 3 2 2 3 0 3 2 0 2 1 3 2 2 0 0 3 0 1 2 0 2]
Prediction: [0 1 3 0 2 3 2 1 0 1 0 3 2 0 3 0 3 2 2 2 2 3 1 2 0 2 1 0 3 1 0 2]



              precision    recall  f1-score   support

       happy       0.89      0.80      0.84        10
     neutral       0.33      0.40      0.36         5
         sad       0.60      0.60      0.60        10
    surprise       0.86      0.86      0.86         7

    accuracy                           0.69        32
   macro avg       0.67      0.66      0.67        32
weighted avg       0.70      0.69      0.69        32





In [37]:
model_name = 'Model 1: Base Neural Network'
save_object(history_base_cnn, f'{MODEL_DIR}/{model_name} - history.pkl')
save_object(metrics_base_cnn, f'{MODEL_DIR}/{model_name} - metrics.pkl')
save_object(model_base_cnn, f'{MODEL_DIR}/{model_name} - models.pkl')
In [38]:
loss_and_accuracy_comparisson(history_base_cnn, data_type='Training Set')
loss_and_accuracy_comparisson(history_base_cnn, data_type='Validation Set')
In [39]:
metrics_base_cnn.style.highlight_max(color = "lightgreen", axis = 0)
Out[39]:
  Accuracy Train Accuracy Val Accuracy Test
Model 1: Base Neural Network (32) - RGB 0.705871 0.719108 0.781250
Model 1: Base Neural Network (32) - GRAY 0.684824 0.702230 0.781250
Model 1: Base Neural Network (64) - RGB 0.711298 0.713281 0.718750
Model 1: Base Neural Network (64) - GRAY 0.708783 0.703838 0.625000
Model 1: Base Neural Network (128) - RGB 0.683235 0.701828 0.750000
Model 1: Base Neural Network (128) - GRAY 0.668145 0.686759 0.687500

⏩ Observations and Insights:

  • In 20 epochs, this base model has reached over 70% of accuracy on the validation set for the majority of the cases (just batch size 128 Grayscale got 68.7%), we can confirm that this model performs well. Only the cases with batch_size 128 do not reach 70% of accuracy. When batch size increase, the model speeds the training process but some of the patterns maybe are not be detected, the model probably has more difficulties capturing the patterns in the data. This is a kind of trade-off between speed and learning.
  • In all cases, the learning curve reveals that the accuracy on the validation set is slightly better than on the training set. This is probably because the data augmentation technique was applied to the training set, and not applied to the validation set. The model learned multiple features based on the changes produced by the data augmentation technique, making the model significantly more robust.
  • Also, the learning curve reflects that model performs similarly in the training and the validation set. Not dealing with overfitting at this level.
  • In this model, it was explored different batch_size: 32, 64, and 128 to analyze how affect the model performance. Batch size 32 gives the best results with both sets (RGB and Grayscale).
  • With batch_size = 64, the model provides similar results to batch_size = 32. A valid trade-off, for a probe of concept, could be to use batch_size = 64, to get similar results but in less time. batch_size = 128 has the worst performance.
  • RGB set performed better, accordingly to the validation set result, in all tested batch_size (32, 64, 128).
  • Analyzing the classification report, it is discovered that Surprise and Happy are the classes with the best accuracy in the majority of the cases. On the other hand, neutral and sad are with the lowest accuracy in those same cases. Probably, the derived patterns from features present in the images for the Happy and Surprise categories are not common in the other categories, making it easier for the models to identify these classes.
  • The best model "Model 1: Base Neural Network (32) - RGB" achieved 71.9% accuracy. RGB provided the best result.

⏩ Recommendations:

  • The learning curve, for "Model 1: Base Neural Network (32) - RGB", seems to be growing, so having more training sessions could be beneficial.
  • Explore other image augmentation techniques to provide more data to the model.
  • A review of the training dataset is necessary to confirm data are good enough.

Model 2: Enhanced Neural Network¶


In [40]:
# Model 2: Enhanced Neural Network
def model_builder_enhanced_cnn(input_shape):
  """Creating an Enhanced Neural Network.
  """
  model = Sequential([
      Conv2D(filters=256, kernel_size=2, activation='relu', padding='same', input_shape=input_shape),
      BatchNormalization(),
      LeakyReLU(0.1),
      MaxPooling2D(pool_size=2),
      Conv2D(filters=128, kernel_size=2, activation='relu', padding='same'),
      BatchNormalization(),
      LeakyReLU(0.1),
      MaxPooling2D(pool_size=2),
      Conv2D(filters=64, kernel_size=2, activation='relu', padding='same'),
      BatchNormalization(),
      LeakyReLU(0.1),
      MaxPooling2D(pool_size=2),
      Conv2D(filters=32, kernel_size=2, activation='relu', padding='same'),
      BatchNormalization(),
      LeakyReLU(0.1),
      MaxPooling2D(pool_size=2),

      Flatten(),
      Dense(512, activation='relu'),
      Dense(128, activation='relu'),
      Dense(4, activation='softmax')
  ])

  # Compile model
  optimizer = Adam(learning_rate = 0.01)        # Using SGD Optimizer
  model.compile(optimizer=optimizer, loss='categorical_crossentropy', metrics=['accuracy'])

  # Generating the summary of the model
  print(model.summary())
  return model
In [41]:
batch_size = 32
model_enhanced_cnn, metrics_enhanced_cnn, history_enhanced_cnn = model_building_and_evaluating_process(
    model_builder_enhanced_cnn, f'Model 2: Enhanced Neural Network ({batch_size})', batch_size=batch_size, epochs=20
)
---------------------------------------------
Model 2: Enhanced Neural Network (32) - RGB
---------------------------------------------
Initializing TF Session and Seed

Data with color_mode=rgb:
Found 15109 images belonging to 4 classes.
Found 127 images belonging to 4 classes.
Found 4977 images belonging to 4 classes.



Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d (Conv2D)             (None, 48, 48, 256)       3328      
                                                                 
 batch_normalization (BatchN  (None, 48, 48, 256)      1024      
 ormalization)                                                   
                                                                 
 leaky_re_lu (LeakyReLU)     (None, 48, 48, 256)       0         
                                                                 
 max_pooling2d (MaxPooling2D  (None, 24, 24, 256)      0         
 )                                                               
                                                                 
 conv2d_1 (Conv2D)           (None, 24, 24, 128)       131200    
                                                                 
 batch_normalization_1 (Batc  (None, 24, 24, 128)      512       
 hNormalization)                                                 
                                                                 
 leaky_re_lu_1 (LeakyReLU)   (None, 24, 24, 128)       0         
                                                                 
 max_pooling2d_1 (MaxPooling  (None, 12, 12, 128)      0         
 2D)                                                             
                                                                 
 conv2d_2 (Conv2D)           (None, 12, 12, 64)        32832     
                                                                 
 batch_normalization_2 (Batc  (None, 12, 12, 64)       256       
 hNormalization)                                                 
                                                                 
 leaky_re_lu_2 (LeakyReLU)   (None, 12, 12, 64)        0         
                                                                 
 max_pooling2d_2 (MaxPooling  (None, 6, 6, 64)         0         
 2D)                                                             
                                                                 
 conv2d_3 (Conv2D)           (None, 6, 6, 32)          8224      
                                                                 
 batch_normalization_3 (Batc  (None, 6, 6, 32)         128       
 hNormalization)                                                 
                                                                 
 leaky_re_lu_3 (LeakyReLU)   (None, 6, 6, 32)          0         
                                                                 
 max_pooling2d_3 (MaxPooling  (None, 3, 3, 32)         0         
 2D)                                                             
                                                                 
 flatten (Flatten)           (None, 288)               0         
                                                                 
 dense (Dense)               (None, 512)               147968    
                                                                 
 dense_1 (Dense)             (None, 128)               65664     
                                                                 
 dense_2 (Dense)             (None, 4)                 516       
                                                                 
=================================================================
Total params: 391,652
Trainable params: 390,692
Non-trainable params: 960
_________________________________________________________________
None



Epoch 1/20
473/473 [==============================] - 69s 141ms/step - loss: 1.3893 - accuracy: 0.3587 - val_loss: 1.3292 - val_accuracy: 0.3757 - lr: 0.0100
Epoch 2/20
473/473 [==============================] - 66s 139ms/step - loss: 1.0473 - accuracy: 0.5200 - val_loss: 1.4469 - val_accuracy: 0.4061 - lr: 0.0100
Epoch 3/20
473/473 [==============================] - 66s 139ms/step - loss: 0.9427 - accuracy: 0.5763 - val_loss: 1.3244 - val_accuracy: 0.4541 - lr: 0.0100
Epoch 4/20
473/473 [==============================] - 66s 140ms/step - loss: 0.8756 - accuracy: 0.6178 - val_loss: 0.8860 - val_accuracy: 0.6295 - lr: 0.0100
Epoch 5/20
473/473 [==============================] - 66s 138ms/step - loss: 0.8234 - accuracy: 0.6549 - val_loss: 0.9004 - val_accuracy: 0.6227 - lr: 0.0100
Epoch 6/20
473/473 [==============================] - 66s 140ms/step - loss: 0.7941 - accuracy: 0.6711 - val_loss: 0.8781 - val_accuracy: 0.6309 - lr: 0.0100
Epoch 7/20
473/473 [==============================] - 66s 140ms/step - loss: 0.7566 - accuracy: 0.6875 - val_loss: 1.0392 - val_accuracy: 0.5425 - lr: 0.0100
Epoch 8/20
473/473 [==============================] - 67s 142ms/step - loss: 0.7365 - accuracy: 0.6935 - val_loss: 0.8052 - val_accuracy: 0.6683 - lr: 0.0100
Epoch 9/20
473/473 [==============================] - 66s 139ms/step - loss: 0.7253 - accuracy: 0.7033 - val_loss: 0.9822 - val_accuracy: 0.5943 - lr: 0.0100
Epoch 10/20
473/473 [==============================] - 66s 140ms/step - loss: 0.6975 - accuracy: 0.7157 - val_loss: 0.7527 - val_accuracy: 0.6962 - lr: 0.0100
Epoch 11/20
473/473 [==============================] - 65s 138ms/step - loss: 0.6905 - accuracy: 0.7151 - val_loss: 1.0202 - val_accuracy: 0.5959 - lr: 0.0100
Epoch 12/20
473/473 [==============================] - 65s 138ms/step - loss: 0.6701 - accuracy: 0.7220 - val_loss: 0.7829 - val_accuracy: 0.6759 - lr: 0.0100
Epoch 13/20
473/473 [==============================] - 66s 139ms/step - loss: 0.6630 - accuracy: 0.7287 - val_loss: 0.8674 - val_accuracy: 0.6582 - lr: 0.0100
Epoch 14/20
473/473 [==============================] - 66s 140ms/step - loss: 0.5698 - accuracy: 0.7699 - val_loss: 0.7297 - val_accuracy: 0.7215 - lr: 0.0020
Epoch 15/20
473/473 [==============================] - 66s 139ms/step - loss: 0.5381 - accuracy: 0.7824 - val_loss: 0.7504 - val_accuracy: 0.7131 - lr: 0.0020
Epoch 16/20
473/473 [==============================] - 66s 140ms/step - loss: 0.5154 - accuracy: 0.7923 - val_loss: 0.7193 - val_accuracy: 0.7271 - lr: 0.0020
Epoch 17/20
473/473 [==============================] - 67s 142ms/step - loss: 0.4999 - accuracy: 0.7931 - val_loss: 0.7519 - val_accuracy: 0.7183 - lr: 0.0020
Epoch 18/20
473/473 [==============================] - 80s 170ms/step - loss: 0.4833 - accuracy: 0.8048 - val_loss: 0.7527 - val_accuracy: 0.7189 - lr: 0.0020
Epoch 19/20
473/473 [==============================] - 79s 166ms/step - loss: 0.4647 - accuracy: 0.8126 - val_loss: 0.7593 - val_accuracy: 0.7265 - lr: 0.0020
Epoch 20/20
473/473 [==============================] - 78s 165ms/step - loss: 0.4299 - accuracy: 0.8293 - val_loss: 0.7631 - val_accuracy: 0.7255 - lr: 4.0000e-04



1/1 [==============================] - 0s 68ms/step - loss: 0.4700 - accuracy: 0.8438
1/1 [==============================] - 0s 180ms/step
Actual    : [0 1 3 0 1 3 2 1 0 2 0 3 2 2 3 0 3 2 0 2 1 3 2 2 0 0 3 0 1 2 0 2]
Prediction: [0 1 3 0 2 3 2 1 0 1 0 3 2 2 3 0 3 2 0 2 1 3 2 2 0 1 2 0 1 1 0 2]



              precision    recall  f1-score   support

       happy       1.00      0.90      0.95        10
     neutral       0.57      0.80      0.67         5
         sad       0.80      0.80      0.80        10
    surprise       1.00      0.86      0.92         7

    accuracy                           0.84        32
   macro avg       0.84      0.84      0.83        32
weighted avg       0.87      0.84      0.85        32





---------------------------------------------
Model 2: Enhanced Neural Network (32) - GRAY
---------------------------------------------
Initializing TF Session and Seed

Data with color_mode=grayscale:
Found 15109 images belonging to 4 classes.
Found 127 images belonging to 4 classes.
Found 4977 images belonging to 4 classes.



Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d (Conv2D)             (None, 48, 48, 256)       1280      
                                                                 
 batch_normalization (BatchN  (None, 48, 48, 256)      1024      
 ormalization)                                                   
                                                                 
 leaky_re_lu (LeakyReLU)     (None, 48, 48, 256)       0         
                                                                 
 max_pooling2d (MaxPooling2D  (None, 24, 24, 256)      0         
 )                                                               
                                                                 
 conv2d_1 (Conv2D)           (None, 24, 24, 128)       131200    
                                                                 
 batch_normalization_1 (Batc  (None, 24, 24, 128)      512       
 hNormalization)                                                 
                                                                 
 leaky_re_lu_1 (LeakyReLU)   (None, 24, 24, 128)       0         
                                                                 
 max_pooling2d_1 (MaxPooling  (None, 12, 12, 128)      0         
 2D)                                                             
                                                                 
 conv2d_2 (Conv2D)           (None, 12, 12, 64)        32832     
                                                                 
 batch_normalization_2 (Batc  (None, 12, 12, 64)       256       
 hNormalization)                                                 
                                                                 
 leaky_re_lu_2 (LeakyReLU)   (None, 12, 12, 64)        0         
                                                                 
 max_pooling2d_2 (MaxPooling  (None, 6, 6, 64)         0         
 2D)                                                             
                                                                 
 conv2d_3 (Conv2D)           (None, 6, 6, 32)          8224      
                                                                 
 batch_normalization_3 (Batc  (None, 6, 6, 32)         128       
 hNormalization)                                                 
                                                                 
 leaky_re_lu_3 (LeakyReLU)   (None, 6, 6, 32)          0         
                                                                 
 max_pooling2d_3 (MaxPooling  (None, 3, 3, 32)         0         
 2D)                                                             
                                                                 
 flatten (Flatten)           (None, 288)               0         
                                                                 
 dense (Dense)               (None, 512)               147968    
                                                                 
 dense_1 (Dense)             (None, 128)               65664     
                                                                 
 dense_2 (Dense)             (None, 4)                 516       
                                                                 
=================================================================
Total params: 389,604
Trainable params: 388,644
Non-trainable params: 960
_________________________________________________________________
None



Epoch 1/20
473/473 [==============================] - 78s 160ms/step - loss: 1.3877 - accuracy: 0.3525 - val_loss: 1.2700 - val_accuracy: 0.3434 - lr: 0.0100
Epoch 2/20
473/473 [==============================] - 72s 153ms/step - loss: 1.0874 - accuracy: 0.5033 - val_loss: 1.1124 - val_accuracy: 0.5079 - lr: 0.0100
Epoch 3/20
473/473 [==============================] - 74s 157ms/step - loss: 0.9372 - accuracy: 0.5949 - val_loss: 0.9057 - val_accuracy: 0.6277 - lr: 0.0100
Epoch 4/20
473/473 [==============================] - 72s 152ms/step - loss: 0.8693 - accuracy: 0.6280 - val_loss: 0.9304 - val_accuracy: 0.6130 - lr: 0.0100
Epoch 5/20
473/473 [==============================] - 69s 146ms/step - loss: 0.8381 - accuracy: 0.6477 - val_loss: 0.9043 - val_accuracy: 0.6299 - lr: 0.0100
Epoch 6/20
473/473 [==============================] - 68s 143ms/step - loss: 0.8052 - accuracy: 0.6638 - val_loss: 1.0157 - val_accuracy: 0.5843 - lr: 0.0100
Epoch 7/20
473/473 [==============================] - 67s 142ms/step - loss: 0.7746 - accuracy: 0.6803 - val_loss: 0.8872 - val_accuracy: 0.6472 - lr: 0.0100
Epoch 8/20
473/473 [==============================] - 67s 141ms/step - loss: 0.7455 - accuracy: 0.6926 - val_loss: 0.8119 - val_accuracy: 0.6723 - lr: 0.0100
Epoch 9/20
473/473 [==============================] - 67s 141ms/step - loss: 0.7258 - accuracy: 0.7029 - val_loss: 1.6141 - val_accuracy: 0.5612 - lr: 0.0100
Epoch 10/20
473/473 [==============================] - 71s 149ms/step - loss: 0.7120 - accuracy: 0.7115 - val_loss: 0.8237 - val_accuracy: 0.6558 - lr: 0.0100
Epoch 11/20
473/473 [==============================] - 69s 146ms/step - loss: 0.6929 - accuracy: 0.7186 - val_loss: 1.1831 - val_accuracy: 0.5001 - lr: 0.0100
Epoch 12/20
473/473 [==============================] - 68s 144ms/step - loss: 0.6094 - accuracy: 0.7522 - val_loss: 0.7039 - val_accuracy: 0.7085 - lr: 0.0020
Epoch 13/20
473/473 [==============================] - 68s 144ms/step - loss: 0.5724 - accuracy: 0.7721 - val_loss: 0.7011 - val_accuracy: 0.7237 - lr: 0.0020
Epoch 14/20
473/473 [==============================] - 71s 149ms/step - loss: 0.5578 - accuracy: 0.7740 - val_loss: 0.7767 - val_accuracy: 0.6958 - lr: 0.0020
Epoch 15/20
473/473 [==============================] - 71s 149ms/step - loss: 0.5400 - accuracy: 0.7831 - val_loss: 0.7318 - val_accuracy: 0.7267 - lr: 0.0020
Epoch 16/20
473/473 [==============================] - 68s 144ms/step - loss: 0.5205 - accuracy: 0.7915 - val_loss: 0.7206 - val_accuracy: 0.7314 - lr: 0.0020
Epoch 17/20
473/473 [==============================] - 69s 146ms/step - loss: 0.4907 - accuracy: 0.8020 - val_loss: 0.7196 - val_accuracy: 0.7304 - lr: 4.0000e-04
Epoch 18/20
473/473 [==============================] - 72s 152ms/step - loss: 0.4783 - accuracy: 0.8084 - val_loss: 0.7350 - val_accuracy: 0.7302 - lr: 4.0000e-04



1/1 [==============================] - 0s 65ms/step - loss: 0.6166 - accuracy: 0.8125
1/1 [==============================] - 0s 172ms/step
Actual    : [0 1 3 0 1 3 2 1 0 2 0 3 2 2 3 0 3 2 0 2 1 3 2 2 0 0 3 0 1 2 0 2]
Prediction: [0 1 3 0 1 3 2 1 0 1 0 3 2 2 3 0 3 2 0 2 2 3 1 2 0 1 1 0 1 1 0 2]



              precision    recall  f1-score   support

       happy       1.00      0.90      0.95        10
     neutral       0.44      0.80      0.57         5
         sad       0.88      0.70      0.78        10
    surprise       1.00      0.86      0.92         7

    accuracy                           0.81        32
   macro avg       0.83      0.81      0.80        32
weighted avg       0.87      0.81      0.83        32





In [42]:
batch_size = 64
model, results, history = model_building_and_evaluating_process(model_builder_enhanced_cnn, f'Model 2: Enhanced Neural Network ({batch_size})',
                                                                batch_size=batch_size, epochs=20)
model_enhanced_cnn.update(model)
metrics_enhanced_cnn = pd.concat([metrics_enhanced_cnn, results])
history_enhanced_cnn.update(history)
---------------------------------------------
Model 2: Enhanced Neural Network (64) - RGB
---------------------------------------------
Initializing TF Session and Seed

Data with color_mode=rgb:
Found 15109 images belonging to 4 classes.
Found 127 images belonging to 4 classes.
Found 4977 images belonging to 4 classes.



Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d (Conv2D)             (None, 48, 48, 256)       3328      
                                                                 
 batch_normalization (BatchN  (None, 48, 48, 256)      1024      
 ormalization)                                                   
                                                                 
 leaky_re_lu (LeakyReLU)     (None, 48, 48, 256)       0         
                                                                 
 max_pooling2d (MaxPooling2D  (None, 24, 24, 256)      0         
 )                                                               
                                                                 
 conv2d_1 (Conv2D)           (None, 24, 24, 128)       131200    
                                                                 
 batch_normalization_1 (Batc  (None, 24, 24, 128)      512       
 hNormalization)                                                 
                                                                 
 leaky_re_lu_1 (LeakyReLU)   (None, 24, 24, 128)       0         
                                                                 
 max_pooling2d_1 (MaxPooling  (None, 12, 12, 128)      0         
 2D)                                                             
                                                                 
 conv2d_2 (Conv2D)           (None, 12, 12, 64)        32832     
                                                                 
 batch_normalization_2 (Batc  (None, 12, 12, 64)       256       
 hNormalization)                                                 
                                                                 
 leaky_re_lu_2 (LeakyReLU)   (None, 12, 12, 64)        0         
                                                                 
 max_pooling2d_2 (MaxPooling  (None, 6, 6, 64)         0         
 2D)                                                             
                                                                 
 conv2d_3 (Conv2D)           (None, 6, 6, 32)          8224      
                                                                 
 batch_normalization_3 (Batc  (None, 6, 6, 32)         128       
 hNormalization)                                                 
                                                                 
 leaky_re_lu_3 (LeakyReLU)   (None, 6, 6, 32)          0         
                                                                 
 max_pooling2d_3 (MaxPooling  (None, 3, 3, 32)         0         
 2D)                                                             
                                                                 
 flatten (Flatten)           (None, 288)               0         
                                                                 
 dense (Dense)               (None, 512)               147968    
                                                                 
 dense_1 (Dense)             (None, 128)               65664     
                                                                 
 dense_2 (Dense)             (None, 4)                 516       
                                                                 
=================================================================
Total params: 391,652
Trainable params: 390,692
Non-trainable params: 960
_________________________________________________________________
None



Epoch 1/20
237/237 [==============================] - 71s 290ms/step - loss: 1.4343 - accuracy: 0.3738 - val_loss: 1.3023 - val_accuracy: 0.4097 - lr: 0.0100
Epoch 2/20
237/237 [==============================] - 68s 287ms/step - loss: 1.0529 - accuracy: 0.5253 - val_loss: 1.1257 - val_accuracy: 0.4959 - lr: 0.0100
Epoch 3/20
237/237 [==============================] - 66s 279ms/step - loss: 0.9480 - accuracy: 0.5859 - val_loss: 1.0692 - val_accuracy: 0.5373 - lr: 0.0100
Epoch 4/20
237/237 [==============================] - 68s 288ms/step - loss: 0.8785 - accuracy: 0.6289 - val_loss: 0.9133 - val_accuracy: 0.6080 - lr: 0.0100
Epoch 5/20
237/237 [==============================] - 66s 278ms/step - loss: 0.8345 - accuracy: 0.6466 - val_loss: 0.9580 - val_accuracy: 0.5672 - lr: 0.0100
Epoch 6/20
237/237 [==============================] - 66s 279ms/step - loss: 0.7965 - accuracy: 0.6662 - val_loss: 1.0283 - val_accuracy: 0.5393 - lr: 0.0100
Epoch 7/20
237/237 [==============================] - 66s 280ms/step - loss: 0.7560 - accuracy: 0.6842 - val_loss: 0.9096 - val_accuracy: 0.6034 - lr: 0.0100
Epoch 8/20
237/237 [==============================] - 67s 282ms/step - loss: 0.7458 - accuracy: 0.6911 - val_loss: 0.9428 - val_accuracy: 0.6223 - lr: 0.0100
Epoch 9/20
237/237 [==============================] - 67s 280ms/step - loss: 0.7233 - accuracy: 0.7022 - val_loss: 0.8124 - val_accuracy: 0.6665 - lr: 0.0100
Epoch 10/20
237/237 [==============================] - 65s 274ms/step - loss: 0.6904 - accuracy: 0.7161 - val_loss: 0.8657 - val_accuracy: 0.6345 - lr: 0.0100
Epoch 11/20
237/237 [==============================] - 64s 269ms/step - loss: 0.6786 - accuracy: 0.7200 - val_loss: 1.0323 - val_accuracy: 0.5746 - lr: 0.0100
Epoch 12/20
237/237 [==============================] - 64s 270ms/step - loss: 0.6735 - accuracy: 0.7235 - val_loss: 0.7623 - val_accuracy: 0.6914 - lr: 0.0100
Epoch 13/20
237/237 [==============================] - 64s 272ms/step - loss: 0.6468 - accuracy: 0.7343 - val_loss: 0.8515 - val_accuracy: 0.6506 - lr: 0.0100
Epoch 14/20
237/237 [==============================] - 66s 280ms/step - loss: 0.6523 - accuracy: 0.7341 - val_loss: 0.8055 - val_accuracy: 0.6735 - lr: 0.0100
Epoch 15/20
237/237 [==============================] - 67s 283ms/step - loss: 0.6228 - accuracy: 0.7449 - val_loss: 0.7623 - val_accuracy: 0.6934 - lr: 0.0100
Epoch 16/20
237/237 [==============================] - 66s 278ms/step - loss: 0.5357 - accuracy: 0.7836 - val_loss: 0.7076 - val_accuracy: 0.7179 - lr: 0.0020
Epoch 17/20
237/237 [==============================] - 67s 281ms/step - loss: 0.5034 - accuracy: 0.7944 - val_loss: 0.7176 - val_accuracy: 0.7195 - lr: 0.0020
Epoch 18/20
237/237 [==============================] - 68s 287ms/step - loss: 0.4814 - accuracy: 0.8036 - val_loss: 0.7248 - val_accuracy: 0.7197 - lr: 0.0020
Epoch 19/20
237/237 [==============================] - 65s 276ms/step - loss: 0.4638 - accuracy: 0.8125 - val_loss: 0.7596 - val_accuracy: 0.7169 - lr: 0.0020
Epoch 20/20
237/237 [==============================] - 66s 278ms/step - loss: 0.4301 - accuracy: 0.8266 - val_loss: 0.7374 - val_accuracy: 0.7368 - lr: 4.0000e-04



1/1 [==============================] - 0s 59ms/step - loss: 0.8335 - accuracy: 0.6875
1/1 [==============================] - 0s 168ms/step
Actual    : [0 1 3 0 1 3 2 1 0 2 0 3 2 2 3 0 3 2 0 2 1 3 2 2 0 0 3 0 1 2 0 2]
Prediction: [0 1 0 0 0 3 2 1 0 2 0 3 2 2 3 0 3 2 0 2 2 2 1 1 0 2 1 0 0 1 0 2]



              precision    recall  f1-score   support

       happy       0.75      0.90      0.82        10
     neutral       0.33      0.40      0.36         5
         sad       0.70      0.70      0.70        10
    surprise       1.00      0.57      0.73         7

    accuracy                           0.69        32
   macro avg       0.70      0.64      0.65        32
weighted avg       0.72      0.69      0.69        32





---------------------------------------------
Model 2: Enhanced Neural Network (64) - GRAY
---------------------------------------------
Initializing TF Session and Seed

Data with color_mode=grayscale:
Found 15109 images belonging to 4 classes.
Found 127 images belonging to 4 classes.
Found 4977 images belonging to 4 classes.



Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d (Conv2D)             (None, 48, 48, 256)       1280      
                                                                 
 batch_normalization (BatchN  (None, 48, 48, 256)      1024      
 ormalization)                                                   
                                                                 
 leaky_re_lu (LeakyReLU)     (None, 48, 48, 256)       0         
                                                                 
 max_pooling2d (MaxPooling2D  (None, 24, 24, 256)      0         
 )                                                               
                                                                 
 conv2d_1 (Conv2D)           (None, 24, 24, 128)       131200    
                                                                 
 batch_normalization_1 (Batc  (None, 24, 24, 128)      512       
 hNormalization)                                                 
                                                                 
 leaky_re_lu_1 (LeakyReLU)   (None, 24, 24, 128)       0         
                                                                 
 max_pooling2d_1 (MaxPooling  (None, 12, 12, 128)      0         
 2D)                                                             
                                                                 
 conv2d_2 (Conv2D)           (None, 12, 12, 64)        32832     
                                                                 
 batch_normalization_2 (Batc  (None, 12, 12, 64)       256       
 hNormalization)                                                 
                                                                 
 leaky_re_lu_2 (LeakyReLU)   (None, 12, 12, 64)        0         
                                                                 
 max_pooling2d_2 (MaxPooling  (None, 6, 6, 64)         0         
 2D)                                                             
                                                                 
 conv2d_3 (Conv2D)           (None, 6, 6, 32)          8224      
                                                                 
 batch_normalization_3 (Batc  (None, 6, 6, 32)         128       
 hNormalization)                                                 
                                                                 
 leaky_re_lu_3 (LeakyReLU)   (None, 6, 6, 32)          0         
                                                                 
 max_pooling2d_3 (MaxPooling  (None, 3, 3, 32)         0         
 2D)                                                             
                                                                 
 flatten (Flatten)           (None, 288)               0         
                                                                 
 dense (Dense)               (None, 512)               147968    
                                                                 
 dense_1 (Dense)             (None, 128)               65664     
                                                                 
 dense_2 (Dense)             (None, 4)                 516       
                                                                 
=================================================================
Total params: 389,604
Trainable params: 388,644
Non-trainable params: 960
_________________________________________________________________
None



Epoch 1/20
237/237 [==============================] - 67s 275ms/step - loss: 1.4771 - accuracy: 0.3600 - val_loss: 1.4708 - val_accuracy: 0.3221 - lr: 0.0100
Epoch 2/20
237/237 [==============================] - 66s 279ms/step - loss: 1.1109 - accuracy: 0.4953 - val_loss: 1.1091 - val_accuracy: 0.4850 - lr: 0.0100
Epoch 3/20
237/237 [==============================] - 67s 281ms/step - loss: 0.9491 - accuracy: 0.5890 - val_loss: 1.3789 - val_accuracy: 0.3982 - lr: 0.0100
Epoch 4/20
237/237 [==============================] - 67s 281ms/step - loss: 0.8670 - accuracy: 0.6362 - val_loss: 1.2985 - val_accuracy: 0.4947 - lr: 0.0100
Epoch 5/20
237/237 [==============================] - 66s 279ms/step - loss: 0.8030 - accuracy: 0.6671 - val_loss: 0.8474 - val_accuracy: 0.6480 - lr: 0.0100
Epoch 6/20
237/237 [==============================] - 65s 275ms/step - loss: 0.7823 - accuracy: 0.6764 - val_loss: 1.2658 - val_accuracy: 0.4623 - lr: 0.0100
Epoch 7/20
237/237 [==============================] - 67s 281ms/step - loss: 0.7453 - accuracy: 0.6904 - val_loss: 1.1134 - val_accuracy: 0.5017 - lr: 0.0100
Epoch 8/20
237/237 [==============================] - 67s 282ms/step - loss: 0.7310 - accuracy: 0.6989 - val_loss: 0.8642 - val_accuracy: 0.6430 - lr: 0.0100
Epoch 9/20
237/237 [==============================] - 67s 284ms/step - loss: 0.6453 - accuracy: 0.7391 - val_loss: 0.7260 - val_accuracy: 0.7060 - lr: 0.0020
Epoch 10/20
237/237 [==============================] - 65s 274ms/step - loss: 0.6114 - accuracy: 0.7509 - val_loss: 0.7420 - val_accuracy: 0.6990 - lr: 0.0020
Epoch 11/20
237/237 [==============================] - 65s 272ms/step - loss: 0.5994 - accuracy: 0.7577 - val_loss: 0.7393 - val_accuracy: 0.6996 - lr: 0.0020
Epoch 12/20
237/237 [==============================] - 66s 279ms/step - loss: 0.5818 - accuracy: 0.7635 - val_loss: 0.7339 - val_accuracy: 0.7071 - lr: 0.0020
Epoch 13/20
237/237 [==============================] - 66s 280ms/step - loss: 0.5456 - accuracy: 0.7798 - val_loss: 0.7099 - val_accuracy: 0.7219 - lr: 4.0000e-04
Epoch 14/20
237/237 [==============================] - 67s 284ms/step - loss: 0.5328 - accuracy: 0.7842 - val_loss: 0.7088 - val_accuracy: 0.7193 - lr: 4.0000e-04
Epoch 15/20
237/237 [==============================] - 69s 290ms/step - loss: 0.5242 - accuracy: 0.7880 - val_loss: 0.7054 - val_accuracy: 0.7277 - lr: 4.0000e-04
Epoch 16/20
237/237 [==============================] - 66s 278ms/step - loss: 0.5188 - accuracy: 0.7888 - val_loss: 0.7322 - val_accuracy: 0.7177 - lr: 4.0000e-04
Epoch 17/20
237/237 [==============================] - 66s 280ms/step - loss: 0.5141 - accuracy: 0.7944 - val_loss: 0.7164 - val_accuracy: 0.7167 - lr: 4.0000e-04
Epoch 18/20
237/237 [==============================] - 65s 273ms/step - loss: 0.5106 - accuracy: 0.7966 - val_loss: 0.7246 - val_accuracy: 0.7227 - lr: 4.0000e-04
Epoch 19/20
237/237 [==============================] - 65s 273ms/step - loss: 0.4973 - accuracy: 0.8042 - val_loss: 0.7274 - val_accuracy: 0.7221 - lr: 8.0000e-05
Epoch 20/20
237/237 [==============================] - 65s 276ms/step - loss: 0.4975 - accuracy: 0.7991 - val_loss: 0.7241 - val_accuracy: 0.7219 - lr: 8.0000e-05



1/1 [==============================] - 0s 59ms/step - loss: 0.5514 - accuracy: 0.7500
1/1 [==============================] - 0s 166ms/step
Actual    : [0 1 3 0 1 3 2 1 0 2 0 3 2 2 3 0 3 2 0 2 1 3 2 2 0 0 3 0 1 2 0 2]
Prediction: [0 1 3 0 2 3 2 1 0 1 0 3 2 2 3 0 3 2 0 2 2 3 1 2 0 1 1 0 0 1 0 2]



              precision    recall  f1-score   support

       happy       0.90      0.90      0.90        10
     neutral       0.29      0.40      0.33         5
         sad       0.78      0.70      0.74        10
    surprise       1.00      0.86      0.92         7

    accuracy                           0.75        32
   macro avg       0.74      0.71      0.72        32
weighted avg       0.79      0.75      0.77        32





In [43]:
batch_size = 128
model, results, history = model_building_and_evaluating_process(model_builder_enhanced_cnn, f'Model 2: Enhanced Neural Network ({batch_size})',
                                                                batch_size=batch_size, epochs=20)
model_enhanced_cnn.update(model)
metrics_enhanced_cnn = pd.concat([metrics_enhanced_cnn, results])
history_enhanced_cnn.update(history)
---------------------------------------------
Model 2: Enhanced Neural Network (128) - RGB
---------------------------------------------
Initializing TF Session and Seed

Data with color_mode=rgb:
Found 15109 images belonging to 4 classes.
Found 127 images belonging to 4 classes.
Found 4977 images belonging to 4 classes.



Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d (Conv2D)             (None, 48, 48, 256)       3328      
                                                                 
 batch_normalization (BatchN  (None, 48, 48, 256)      1024      
 ormalization)                                                   
                                                                 
 leaky_re_lu (LeakyReLU)     (None, 48, 48, 256)       0         
                                                                 
 max_pooling2d (MaxPooling2D  (None, 24, 24, 256)      0         
 )                                                               
                                                                 
 conv2d_1 (Conv2D)           (None, 24, 24, 128)       131200    
                                                                 
 batch_normalization_1 (Batc  (None, 24, 24, 128)      512       
 hNormalization)                                                 
                                                                 
 leaky_re_lu_1 (LeakyReLU)   (None, 24, 24, 128)       0         
                                                                 
 max_pooling2d_1 (MaxPooling  (None, 12, 12, 128)      0         
 2D)                                                             
                                                                 
 conv2d_2 (Conv2D)           (None, 12, 12, 64)        32832     
                                                                 
 batch_normalization_2 (Batc  (None, 12, 12, 64)       256       
 hNormalization)                                                 
                                                                 
 leaky_re_lu_2 (LeakyReLU)   (None, 12, 12, 64)        0         
                                                                 
 max_pooling2d_2 (MaxPooling  (None, 6, 6, 64)         0         
 2D)                                                             
                                                                 
 conv2d_3 (Conv2D)           (None, 6, 6, 32)          8224      
                                                                 
 batch_normalization_3 (Batc  (None, 6, 6, 32)         128       
 hNormalization)                                                 
                                                                 
 leaky_re_lu_3 (LeakyReLU)   (None, 6, 6, 32)          0         
                                                                 
 max_pooling2d_3 (MaxPooling  (None, 3, 3, 32)         0         
 2D)                                                             
                                                                 
 flatten (Flatten)           (None, 288)               0         
                                                                 
 dense (Dense)               (None, 512)               147968    
                                                                 
 dense_1 (Dense)             (None, 128)               65664     
                                                                 
 dense_2 (Dense)             (None, 4)                 516       
                                                                 
=================================================================
Total params: 391,652
Trainable params: 390,692
Non-trainable params: 960
_________________________________________________________________
None



Epoch 1/20
119/119 [==============================] - 64s 519ms/step - loss: 1.6542 - accuracy: 0.3290 - val_loss: 2.4356 - val_accuracy: 0.2767 - lr: 0.0100
Epoch 2/20
119/119 [==============================] - 61s 512ms/step - loss: 1.1971 - accuracy: 0.4397 - val_loss: 1.4953 - val_accuracy: 0.3705 - lr: 0.0100
Epoch 3/20
119/119 [==============================] - 61s 515ms/step - loss: 0.9932 - accuracy: 0.5528 - val_loss: 1.2061 - val_accuracy: 0.4422 - lr: 0.0100
Epoch 4/20
119/119 [==============================] - 64s 536ms/step - loss: 0.9217 - accuracy: 0.6010 - val_loss: 1.1810 - val_accuracy: 0.4862 - lr: 0.0100
Epoch 5/20
119/119 [==============================] - 63s 527ms/step - loss: 0.8618 - accuracy: 0.6308 - val_loss: 1.0730 - val_accuracy: 0.5421 - lr: 0.0100
Epoch 6/20
119/119 [==============================] - 62s 522ms/step - loss: 0.8079 - accuracy: 0.6564 - val_loss: 1.1524 - val_accuracy: 0.5228 - lr: 0.0100
Epoch 7/20
119/119 [==============================] - 61s 511ms/step - loss: 0.7682 - accuracy: 0.6809 - val_loss: 0.8772 - val_accuracy: 0.6488 - lr: 0.0100
Epoch 8/20
119/119 [==============================] - 62s 519ms/step - loss: 0.7431 - accuracy: 0.6910 - val_loss: 0.8538 - val_accuracy: 0.6484 - lr: 0.0100
Epoch 9/20
119/119 [==============================] - 62s 519ms/step - loss: 0.7175 - accuracy: 0.7035 - val_loss: 0.9057 - val_accuracy: 0.6367 - lr: 0.0100
Epoch 10/20
119/119 [==============================] - 61s 515ms/step - loss: 0.6923 - accuracy: 0.7154 - val_loss: 0.8409 - val_accuracy: 0.6381 - lr: 0.0100
Epoch 11/20
119/119 [==============================] - 61s 508ms/step - loss: 0.6918 - accuracy: 0.7132 - val_loss: 1.1288 - val_accuracy: 0.5558 - lr: 0.0100
Epoch 12/20
119/119 [==============================] - 61s 515ms/step - loss: 0.6682 - accuracy: 0.7259 - val_loss: 0.7400 - val_accuracy: 0.7083 - lr: 0.0100
Epoch 13/20
119/119 [==============================] - 62s 518ms/step - loss: 0.6393 - accuracy: 0.7362 - val_loss: 1.1642 - val_accuracy: 0.5188 - lr: 0.0100
Epoch 14/20
119/119 [==============================] - 62s 522ms/step - loss: 0.6471 - accuracy: 0.7364 - val_loss: 0.8443 - val_accuracy: 0.6552 - lr: 0.0100
Epoch 15/20
119/119 [==============================] - 63s 528ms/step - loss: 0.6339 - accuracy: 0.7405 - val_loss: 1.0644 - val_accuracy: 0.5449 - lr: 0.0100
Epoch 16/20
119/119 [==============================] - 65s 548ms/step - loss: 0.5522 - accuracy: 0.7772 - val_loss: 0.7175 - val_accuracy: 0.7185 - lr: 0.0020
Epoch 17/20
119/119 [==============================] - 63s 533ms/step - loss: 0.5399 - accuracy: 0.7806 - val_loss: 0.7433 - val_accuracy: 0.7203 - lr: 0.0020
Epoch 18/20
119/119 [==============================] - 63s 525ms/step - loss: 0.5165 - accuracy: 0.7950 - val_loss: 0.7341 - val_accuracy: 0.7171 - lr: 0.0020
Epoch 19/20
119/119 [==============================] - 63s 530ms/step - loss: 0.4966 - accuracy: 0.7983 - val_loss: 0.7138 - val_accuracy: 0.7292 - lr: 0.0020
Epoch 20/20
119/119 [==============================] - 63s 529ms/step - loss: 0.4891 - accuracy: 0.8049 - val_loss: 0.8095 - val_accuracy: 0.7024 - lr: 0.0020



1/1 [==============================] - 0s 58ms/step - loss: 0.7810 - accuracy: 0.7812
1/1 [==============================] - 0s 166ms/step
Actual    : [0 1 3 0 1 3 2 1 0 2 0 3 2 2 3 0 3 2 0 2 1 3 2 2 0 0 3 0 1 2 0 2]
Prediction: [0 2 3 0 1 3 2 1 0 1 0 3 2 2 3 0 3 2 0 2 2 2 0 2 0 0 1 0 0 2 0 2]



              precision    recall  f1-score   support

       happy       0.83      1.00      0.91        10
     neutral       0.50      0.40      0.44         5
         sad       0.73      0.80      0.76        10
    surprise       1.00      0.71      0.83         7

    accuracy                           0.78        32
   macro avg       0.77      0.73      0.74        32
weighted avg       0.78      0.78      0.77        32





---------------------------------------------
Model 2: Enhanced Neural Network (128) - GRAY
---------------------------------------------
Initializing TF Session and Seed

Data with color_mode=grayscale:
Found 15109 images belonging to 4 classes.
Found 127 images belonging to 4 classes.
Found 4977 images belonging to 4 classes.



Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d (Conv2D)             (None, 48, 48, 256)       1280      
                                                                 
 batch_normalization (BatchN  (None, 48, 48, 256)      1024      
 ormalization)                                                   
                                                                 
 leaky_re_lu (LeakyReLU)     (None, 48, 48, 256)       0         
                                                                 
 max_pooling2d (MaxPooling2D  (None, 24, 24, 256)      0         
 )                                                               
                                                                 
 conv2d_1 (Conv2D)           (None, 24, 24, 128)       131200    
                                                                 
 batch_normalization_1 (Batc  (None, 24, 24, 128)      512       
 hNormalization)                                                 
                                                                 
 leaky_re_lu_1 (LeakyReLU)   (None, 24, 24, 128)       0         
                                                                 
 max_pooling2d_1 (MaxPooling  (None, 12, 12, 128)      0         
 2D)                                                             
                                                                 
 conv2d_2 (Conv2D)           (None, 12, 12, 64)        32832     
                                                                 
 batch_normalization_2 (Batc  (None, 12, 12, 64)       256       
 hNormalization)                                                 
                                                                 
 leaky_re_lu_2 (LeakyReLU)   (None, 12, 12, 64)        0         
                                                                 
 max_pooling2d_2 (MaxPooling  (None, 6, 6, 64)         0         
 2D)                                                             
                                                                 
 conv2d_3 (Conv2D)           (None, 6, 6, 32)          8224      
                                                                 
 batch_normalization_3 (Batc  (None, 6, 6, 32)         128       
 hNormalization)                                                 
                                                                 
 leaky_re_lu_3 (LeakyReLU)   (None, 6, 6, 32)          0         
                                                                 
 max_pooling2d_3 (MaxPooling  (None, 3, 3, 32)         0         
 2D)                                                             
                                                                 
 flatten (Flatten)           (None, 288)               0         
                                                                 
 dense (Dense)               (None, 512)               147968    
                                                                 
 dense_1 (Dense)             (None, 128)               65664     
                                                                 
 dense_2 (Dense)             (None, 4)                 516       
                                                                 
=================================================================
Total params: 389,604
Trainable params: 388,644
Non-trainable params: 960
_________________________________________________________________
None



Epoch 1/20
119/119 [==============================] - 66s 538ms/step - loss: 1.6232 - accuracy: 0.3075 - val_loss: 1.3630 - val_accuracy: 0.3719 - lr: 0.0100
Epoch 2/20
119/119 [==============================] - 63s 531ms/step - loss: 1.3055 - accuracy: 0.3508 - val_loss: 1.3287 - val_accuracy: 0.3661 - lr: 0.0100
Epoch 3/20
119/119 [==============================] - 63s 528ms/step - loss: 1.1279 - accuracy: 0.4720 - val_loss: 1.7102 - val_accuracy: 0.2885 - lr: 0.0100
Epoch 4/20
119/119 [==============================] - 63s 530ms/step - loss: 0.9815 - accuracy: 0.5736 - val_loss: 1.1245 - val_accuracy: 0.4917 - lr: 0.0100
Epoch 5/20
119/119 [==============================] - 64s 535ms/step - loss: 0.8900 - accuracy: 0.6237 - val_loss: 1.3265 - val_accuracy: 0.4374 - lr: 0.0100
Epoch 6/20
119/119 [==============================] - 63s 527ms/step - loss: 0.8142 - accuracy: 0.6593 - val_loss: 0.9534 - val_accuracy: 0.5994 - lr: 0.0100
Epoch 7/20
119/119 [==============================] - 62s 524ms/step - loss: 0.7780 - accuracy: 0.6763 - val_loss: 0.8655 - val_accuracy: 0.6317 - lr: 0.0100
Epoch 8/20
119/119 [==============================] - 62s 524ms/step - loss: 0.7418 - accuracy: 0.6944 - val_loss: 0.9340 - val_accuracy: 0.6219 - lr: 0.0100
Epoch 9/20
119/119 [==============================] - 62s 521ms/step - loss: 0.7574 - accuracy: 0.6847 - val_loss: 0.9271 - val_accuracy: 0.6259 - lr: 0.0100
Epoch 10/20
119/119 [==============================] - 62s 522ms/step - loss: 0.7046 - accuracy: 0.7081 - val_loss: 1.6188 - val_accuracy: 0.4278 - lr: 0.0100
Epoch 11/20
119/119 [==============================] - 63s 527ms/step - loss: 0.6439 - accuracy: 0.7373 - val_loss: 0.7312 - val_accuracy: 0.7091 - lr: 0.0020
Epoch 12/20
119/119 [==============================] - 63s 530ms/step - loss: 0.6138 - accuracy: 0.7464 - val_loss: 0.7014 - val_accuracy: 0.7165 - lr: 0.0020
Epoch 13/20
119/119 [==============================] - 63s 532ms/step - loss: 0.5905 - accuracy: 0.7578 - val_loss: 0.7234 - val_accuracy: 0.7133 - lr: 0.0020
Epoch 14/20
119/119 [==============================] - 63s 529ms/step - loss: 0.5896 - accuracy: 0.7609 - val_loss: 0.7147 - val_accuracy: 0.7187 - lr: 0.0020
Epoch 15/20
119/119 [==============================] - 62s 524ms/step - loss: 0.5730 - accuracy: 0.7642 - val_loss: 0.7030 - val_accuracy: 0.7227 - lr: 0.0020
Epoch 16/20
119/119 [==============================] - 62s 524ms/step - loss: 0.5514 - accuracy: 0.7771 - val_loss: 0.7115 - val_accuracy: 0.7217 - lr: 4.0000e-04
Epoch 17/20
119/119 [==============================] - 62s 525ms/step - loss: 0.5377 - accuracy: 0.7799 - val_loss: 0.7362 - val_accuracy: 0.7153 - lr: 4.0000e-04



1/1 [==============================] - 0s 60ms/step - loss: 0.7239 - accuracy: 0.7500
1/1 [==============================] - 0s 162ms/step
Actual    : [0 1 3 0 1 3 2 1 0 2 0 3 2 2 3 0 3 2 0 2 1 3 2 2 0 0 3 0 1 2 0 2]
Prediction: [0 1 3 0 0 3 2 1 0 1 0 3 2 0 3 0 3 2 0 2 2 3 1 1 0 0 1 0 1 0 0 2]



              precision    recall  f1-score   support

       happy       0.77      1.00      0.87        10
     neutral       0.43      0.60      0.50         5
         sad       0.83      0.50      0.62        10
    surprise       1.00      0.86      0.92         7

    accuracy                           0.75        32
   macro avg       0.76      0.74      0.73        32
weighted avg       0.79      0.75      0.75        32





In [44]:
model_name = 'Model 2: Enhanced Neural Network'
save_object(history_enhanced_cnn, f'{MODEL_DIR}/{model_name} - history.pkl')
save_object(metrics_enhanced_cnn, f'{MODEL_DIR}/{model_name} - metrics.pkl')
save_object(model_enhanced_cnn, f'{MODEL_DIR}/{model_name} - models.pkl')
In [45]:
loss_and_accuracy_comparisson(history_enhanced_cnn, data_type='Training Set')
loss_and_accuracy_comparisson(history_enhanced_cnn, data_type='Validation Set')
In [46]:
metrics_enhanced_cnn.style.highlight_max(color = "lightgreen", axis = 0)
Out[46]:
  Accuracy Train Accuracy Val Accuracy Test
Model 2: Enhanced Neural Network (32) - RGB 0.792309 0.727145 0.843750
Model 2: Enhanced Neural Network (32) - GRAY 0.791515 0.731364 0.812500
Model 2: Enhanced Neural Network (64) - RGB 0.826593 0.736789 0.687500
Model 2: Enhanced Neural Network (64) - GRAY 0.788007 0.727748 0.750000
Model 2: Enhanced Neural Network (128) - RGB 0.798332 0.729154 0.781250
Model 2: Enhanced Neural Network (128) - GRAY 0.764180 0.722725 0.750000

⏩ Observations and Insights:

  • This second model, a kind of enhanced neural network, provided better results (73.7% accuracy in the best case and 72.3% in the worst case), however, the learning curve shows overfitting. Probably adding some Dropout layers will help us to deal with this.
  • In this model, it is observed again that batch_size 32 and 64 are the best models. On the next prepared models, batch_size = 32 will be the only selected option, as it is confirmed that this model gives a consistently good result.
  • The classification report provides interesting facts: Surprise and Happy are the best evaluated classess, reaching more than 90% in the majority of cases. Sad and Neutral are the classes with the lower accuracy. The complexity of this model is detecting better the features of each class and at the same time learning the noise of the training set. This overfitting needs to be treated, but this model looks promising.
  • The best model "Model 2: Enhanced Neural Network (64) - RGB" achieved 73.7% accuracy. RGB set provided the best result.

⏩ Recommendations:

  • Technique to deal with the overfitting needs to be applied to the model "Model 2: Enhanced Neural Network (64) - RGB" and see if this improves the result in the validation set.
  • Explore other image augmentation techniques to provide more data to the model.
  • A review of the training dataset is necessary to confirm data are good enough.

Think About It:

  • Did the models have a satisfactory performance? If not, then what are the possible reasons?
  • Which Color mode showed better overall performance? What are the possible reasons? Do you think having 'rgb' color mode is needed because the images are already black and white?

⏩ Observations and Insights:

  • Comparing the performance of the models so far ("Model 1: Base Neural Network (64) - GRAY" and "Model 2: Enhanced Neural Network (32) - RGB"), they are having a good performance, 70.8%, and 73.6% respectively. However the second model shows overfitting, maybe some regularization technique will be necessary (oversampling, dropout, batchnormalization, etc).
  • In "Model 1: Base Neural Network (64) - GRAY" the Grayscale set provides the best result. In "Model 2: Enhanced Neural Network (32) - RGB" RGB provides the best result, however, the difference with the grayscale set is not big enough.
  • In regards to which color mode could be better, will depend on the data itself and on the model that will be applied. If our data are grayscale images encoded in one channel, transforming it to have RGB will not provide any additional meaningful information to the model. Color always means more computational resources, but some models work better with them. This can be a kind of trade-off between speed and accuracy learning.

Transfer Learning Architectures¶

In this section, we will create several Transfer Learning architectures. For the pre-trained models, we will select three popular architectures namely, VGG16, ResNet v2, and Efficient Net. The difference between these architectures and the previous architectures is that these will require 3 input channels while the earlier ones worked on 'grayscale' images. Therefore, we need to create new DataLoaders.

Creating our Data Loaders for Transfer Learning Architectures¶

In this section, we are creating data loaders that we will use as inputs to our Neural Network. We will have to go with color_mode = 'rgb' as this is the required format for the transfer learning architectures.

In [47]:
train_generator_rgb, _, _ = create_data_loaders(IMG_DIR, 'rgb')

# Taking a look at some examples of our augmented training data with color_mode='rgb'.
images, labels = next(train_generator_rgb)
fig, axes = plt.subplots(1, 8, figsize = (10, 2))
for (image, label, ax) in zip(images, labels, axes.flatten()):
  ax.imshow(image)
  ax.set_title(FACE_EXPRESSIONS[list(label).index(1)], fontsize=8)
  ax.axis('off')
plt.show()
Data with color_mode=rgb:
Found 15109 images belonging to 4 classes.
Found 127 images belonging to 4 classes.
Found 4977 images belonging to 4 classes.

Model 3: VGG16¶


In [48]:
VGG16(weights='imagenet', include_top=False, input_shape = (48, 48, 3)).summary()
Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/vgg16/vgg16_weights_tf_dim_ordering_tf_kernels_notop.h5
58889256/58889256 [==============================] - 0s 0us/step
Model: "vgg16"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_1 (InputLayer)        [(None, 48, 48, 3)]       0         
                                                                 
 block1_conv1 (Conv2D)       (None, 48, 48, 64)        1792      
                                                                 
 block1_conv2 (Conv2D)       (None, 48, 48, 64)        36928     
                                                                 
 block1_pool (MaxPooling2D)  (None, 24, 24, 64)        0         
                                                                 
 block2_conv1 (Conv2D)       (None, 24, 24, 128)       73856     
                                                                 
 block2_conv2 (Conv2D)       (None, 24, 24, 128)       147584    
                                                                 
 block2_pool (MaxPooling2D)  (None, 12, 12, 128)       0         
                                                                 
 block3_conv1 (Conv2D)       (None, 12, 12, 256)       295168    
                                                                 
 block3_conv2 (Conv2D)       (None, 12, 12, 256)       590080    
                                                                 
 block3_conv3 (Conv2D)       (None, 12, 12, 256)       590080    
                                                                 
 block3_pool (MaxPooling2D)  (None, 6, 6, 256)         0         
                                                                 
 block4_conv1 (Conv2D)       (None, 6, 6, 512)         1180160   
                                                                 
 block4_conv2 (Conv2D)       (None, 6, 6, 512)         2359808   
                                                                 
 block4_conv3 (Conv2D)       (None, 6, 6, 512)         2359808   
                                                                 
 block4_pool (MaxPooling2D)  (None, 3, 3, 512)         0         
                                                                 
 block5_conv1 (Conv2D)       (None, 3, 3, 512)         2359808   
                                                                 
 block5_conv2 (Conv2D)       (None, 3, 3, 512)         2359808   
                                                                 
 block5_conv3 (Conv2D)       (None, 3, 3, 512)         2359808   
                                                                 
 block5_pool (MaxPooling2D)  (None, 1, 1, 512)         0         
                                                                 
=================================================================
Total params: 14,714,688
Trainable params: 14,714,688
Non-trainable params: 0
_________________________________________________________________
In [49]:
# Model 3: VGG16
def model_builder_vgg16(input_shape, tune=0):
  """Transfer approach with the VGG16 Model
  """
  vgg_model = VGG16(weights='imagenet', include_top=False, input_shape = input_shape)
  # Making all the layers of the VGG model non-trainable
  if tune > 0:
    for layer in vgg_model.layers[:-tune]:
        layer.trainable = False
  else:
    for layer in vgg_model.layers:
        layer.trainable = False

  model = Sequential([
      vgg_model,
      Flatten(),
      Dense(256, activation='relu'),
      Dense(128, activation='relu'),
      Dropout(0.3),
      Dense(64, activation='relu'),
      BatchNormalization(),
      Dense(4, activation='softmax')
  ])

  # Compile model
  optimizer = Adam(learning_rate = 0.001)        # Using SGD Optimizer
  model.compile(optimizer=optimizer, loss='categorical_crossentropy', metrics=['accuracy'])

  # Generating the summary of the model
  print(model.summary())
  return model
In [50]:
batch_size = 32
model_vgg16, metric_vgg16, history_vgg16 = model_building_and_evaluating_process(
    model_builder_vgg16, f'Model 3: VGG16 ({batch_size})', batch_size=batch_size, epochs=20, include_grayscale=False
)
# model_vgg16['Model 3: VGG16 (32) - RGB'].get_metrics_result()['accuracy'].numpy()
---------------------------------------------
Model 3: VGG16 (32) - RGB
---------------------------------------------
Initializing TF Session and Seed

Data with color_mode=rgb:
Found 15109 images belonging to 4 classes.
Found 127 images belonging to 4 classes.
Found 4977 images belonging to 4 classes.



Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 vgg16 (Functional)          (None, 1, 1, 512)         14714688  
                                                                 
 flatten (Flatten)           (None, 512)               0         
                                                                 
 dense (Dense)               (None, 256)               131328    
                                                                 
 dense_1 (Dense)             (None, 128)               32896     
                                                                 
 dropout (Dropout)           (None, 128)               0         
                                                                 
 dense_2 (Dense)             (None, 64)                8256      
                                                                 
 batch_normalization (BatchN  (None, 64)               256       
 ormalization)                                                   
                                                                 
 dense_3 (Dense)             (None, 4)                 260       
                                                                 
=================================================================
Total params: 14,887,684
Trainable params: 172,868
Non-trainable params: 14,714,816
_________________________________________________________________
None



Epoch 1/20
473/473 [==============================] - 41s 83ms/step - loss: 1.2524 - accuracy: 0.4357 - val_loss: 1.0932 - val_accuracy: 0.5252 - lr: 0.0010
Epoch 2/20
473/473 [==============================] - 38s 80ms/step - loss: 1.1323 - accuracy: 0.5042 - val_loss: 1.2063 - val_accuracy: 0.4663 - lr: 0.0010
Epoch 3/20
473/473 [==============================] - 37s 79ms/step - loss: 1.0921 - accuracy: 0.5343 - val_loss: 1.1006 - val_accuracy: 0.5186 - lr: 0.0010
Epoch 4/20
473/473 [==============================] - 39s 82ms/step - loss: 1.0750 - accuracy: 0.5391 - val_loss: 1.0562 - val_accuracy: 0.5431 - lr: 0.0010
Epoch 5/20
473/473 [==============================] - 38s 80ms/step - loss: 1.0516 - accuracy: 0.5501 - val_loss: 1.0475 - val_accuracy: 0.5479 - lr: 0.0010
Epoch 6/20
473/473 [==============================] - 38s 80ms/step - loss: 1.0294 - accuracy: 0.5634 - val_loss: 1.0337 - val_accuracy: 0.5588 - lr: 0.0010
Epoch 7/20
473/473 [==============================] - 37s 79ms/step - loss: 1.0172 - accuracy: 0.5627 - val_loss: 1.1040 - val_accuracy: 0.5236 - lr: 0.0010
Epoch 8/20
473/473 [==============================] - 37s 78ms/step - loss: 1.0022 - accuracy: 0.5729 - val_loss: 1.0965 - val_accuracy: 0.5250 - lr: 0.0010
Epoch 9/20
473/473 [==============================] - 37s 78ms/step - loss: 0.9757 - accuracy: 0.5891 - val_loss: 1.0464 - val_accuracy: 0.5437 - lr: 0.0010
Epoch 10/20
473/473 [==============================] - 38s 80ms/step - loss: 0.9240 - accuracy: 0.6129 - val_loss: 1.0171 - val_accuracy: 0.5668 - lr: 2.0000e-04
Epoch 11/20
473/473 [==============================] - 37s 79ms/step - loss: 0.8960 - accuracy: 0.6282 - val_loss: 1.0418 - val_accuracy: 0.5531 - lr: 2.0000e-04
Epoch 12/20
473/473 [==============================] - 37s 78ms/step - loss: 0.8756 - accuracy: 0.6335 - val_loss: 1.0299 - val_accuracy: 0.5576 - lr: 2.0000e-04
Epoch 13/20
473/473 [==============================] - 37s 79ms/step - loss: 0.8700 - accuracy: 0.6419 - val_loss: 1.0422 - val_accuracy: 0.5622 - lr: 2.0000e-04
Epoch 14/20
473/473 [==============================] - 38s 80ms/step - loss: 0.8420 - accuracy: 0.6513 - val_loss: 1.0207 - val_accuracy: 0.5732 - lr: 4.0000e-05
Epoch 15/20
473/473 [==============================] - 37s 78ms/step - loss: 0.8319 - accuracy: 0.6550 - val_loss: 1.0324 - val_accuracy: 0.5644 - lr: 4.0000e-05



1/1 [==============================] - 0s 71ms/step - loss: 0.8450 - accuracy: 0.6250
1/1 [==============================] - 0s 225ms/step
Actual    : [0 1 3 0 1 3 2 1 0 2 0 3 2 2 3 0 3 2 0 2 1 3 2 2 0 0 3 0 1 2 0 2]
Prediction: [0 2 0 1 2 3 1 1 0 2 1 3 2 2 3 0 3 2 0 2 1 3 1 2 1 2 1 0 2 1 0 2]



              precision    recall  f1-score   support

       happy       0.86      0.60      0.71        10
     neutral       0.22      0.40      0.29         5
         sad       0.64      0.70      0.67        10
    surprise       1.00      0.71      0.83         7

    accuracy                           0.62        32
   macro avg       0.68      0.60      0.62        32
weighted avg       0.72      0.62      0.66        32





In [51]:
# Let's try again tuning some of the last layers in the transfer model
batch_size = 32
model, results, history = model_building_and_evaluating_process(model_builder_vgg16, f'Model 3: Tuned VGG16 ({batch_size})',
                                                                batch_size=batch_size, epochs=20, tune=4, include_grayscale=False)
model_vgg16.update(model)
metric_vgg16 = pd.concat([metric_vgg16, results])
history_vgg16.update(history)
---------------------------------------------
Model 3: Tuned VGG16 (32) - RGB
---------------------------------------------
Initializing TF Session and Seed

Data with color_mode=rgb:
Found 15109 images belonging to 4 classes.
Found 127 images belonging to 4 classes.
Found 4977 images belonging to 4 classes.



Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 vgg16 (Functional)          (None, 1, 1, 512)         14714688  
                                                                 
 flatten (Flatten)           (None, 512)               0         
                                                                 
 dense (Dense)               (None, 256)               131328    
                                                                 
 dense_1 (Dense)             (None, 128)               32896     
                                                                 
 dropout (Dropout)           (None, 128)               0         
                                                                 
 dense_2 (Dense)             (None, 64)                8256      
                                                                 
 batch_normalization (BatchN  (None, 64)               256       
 ormalization)                                                   
                                                                 
 dense_3 (Dense)             (None, 4)                 260       
                                                                 
=================================================================
Total params: 14,887,684
Trainable params: 7,252,292
Non-trainable params: 7,635,392
_________________________________________________________________
None



Epoch 1/20
473/473 [==============================] - 60s 123ms/step - loss: 1.1853 - accuracy: 0.4585 - val_loss: 0.9170 - val_accuracy: 0.5789 - lr: 0.0010
Epoch 2/20
473/473 [==============================] - 58s 122ms/step - loss: 1.0514 - accuracy: 0.5244 - val_loss: 0.9705 - val_accuracy: 0.5971 - lr: 0.0010
Epoch 3/20
473/473 [==============================] - 58s 122ms/step - loss: 0.9214 - accuracy: 0.5940 - val_loss: 0.8417 - val_accuracy: 0.6438 - lr: 0.0010
Epoch 4/20
473/473 [==============================] - 56s 118ms/step - loss: 0.8536 - accuracy: 0.6485 - val_loss: 0.9996 - val_accuracy: 0.5863 - lr: 0.0010
Epoch 5/20
473/473 [==============================] - 57s 121ms/step - loss: 0.8037 - accuracy: 0.6740 - val_loss: 0.8071 - val_accuracy: 0.6713 - lr: 0.0010
Epoch 6/20
473/473 [==============================] - 57s 121ms/step - loss: 0.7712 - accuracy: 0.6879 - val_loss: 0.7755 - val_accuracy: 0.6769 - lr: 0.0010
Epoch 7/20
473/473 [==============================] - 58s 122ms/step - loss: 0.7429 - accuracy: 0.7063 - val_loss: 0.7747 - val_accuracy: 0.6799 - lr: 0.0010
Epoch 8/20
473/473 [==============================] - 56s 119ms/step - loss: 0.7550 - accuracy: 0.6960 - val_loss: 0.9831 - val_accuracy: 0.5990 - lr: 0.0010
Epoch 9/20
473/473 [==============================] - 56s 118ms/step - loss: 0.7480 - accuracy: 0.7009 - val_loss: 1.0221 - val_accuracy: 0.5720 - lr: 0.0010
Epoch 10/20
473/473 [==============================] - 57s 120ms/step - loss: 0.7018 - accuracy: 0.7259 - val_loss: 0.7804 - val_accuracy: 0.6888 - lr: 0.0010
Epoch 11/20
473/473 [==============================] - 57s 121ms/step - loss: 0.6038 - accuracy: 0.7641 - val_loss: 0.7710 - val_accuracy: 0.7022 - lr: 2.0000e-04
Epoch 12/20
473/473 [==============================] - 58s 122ms/step - loss: 0.5686 - accuracy: 0.7844 - val_loss: 0.7590 - val_accuracy: 0.7129 - lr: 2.0000e-04
Epoch 13/20
473/473 [==============================] - 56s 119ms/step - loss: 0.5485 - accuracy: 0.7941 - val_loss: 0.7905 - val_accuracy: 0.6954 - lr: 2.0000e-04
Epoch 14/20
473/473 [==============================] - 56s 119ms/step - loss: 0.5202 - accuracy: 0.8088 - val_loss: 0.8029 - val_accuracy: 0.7022 - lr: 2.0000e-04
Epoch 15/20
473/473 [==============================] - 56s 118ms/step - loss: 0.5028 - accuracy: 0.8125 - val_loss: 0.8014 - val_accuracy: 0.7026 - lr: 2.0000e-04
Epoch 16/20
473/473 [==============================] - 56s 119ms/step - loss: 0.4603 - accuracy: 0.8345 - val_loss: 0.8063 - val_accuracy: 0.7016 - lr: 4.0000e-05
Epoch 17/20
473/473 [==============================] - 56s 119ms/step - loss: 0.4422 - accuracy: 0.8435 - val_loss: 0.8098 - val_accuracy: 0.7048 - lr: 4.0000e-05



1/1 [==============================] - 0s 78ms/step - loss: 0.5211 - accuracy: 0.8125
1/1 [==============================] - 0s 240ms/step
Actual    : [0 1 3 0 1 3 2 1 0 2 0 3 2 2 3 0 3 2 0 2 1 3 2 2 0 0 3 0 1 2 0 2]
Prediction: [0 1 3 0 1 3 1 1 0 2 0 3 2 2 3 0 3 2 0 2 1 3 0 1 0 1 1 0 1 1 0 2]



              precision    recall  f1-score   support

       happy       0.90      0.90      0.90        10
     neutral       0.50      1.00      0.67         5
         sad       1.00      0.60      0.75        10
    surprise       1.00      0.86      0.92         7

    accuracy                           0.81        32
   macro avg       0.85      0.84      0.81        32
weighted avg       0.89      0.81      0.82        32





In [52]:
model_name = 'Model 3: VGG16'
save_object(history_vgg16, f'{MODEL_DIR}/{model_name} - history.pkl')
save_object(metric_vgg16, f'{MODEL_DIR}/{model_name} - metrics.pkl')
save_object(model_vgg16, f'{MODEL_DIR}/{model_name} - models.pkl')
In [53]:
loss_and_accuracy_comparisson(history_vgg16, data_type='Training Set')
loss_and_accuracy_comparisson(history_vgg16, data_type='Validation Set')
In [54]:
metric_vgg16.style.highlight_max(color = "lightgreen", axis = 0)
Out[54]:
  Accuracy Train Accuracy Val Accuracy Test
Model 3: VGG16 (32) - RGB 0.651267 0.573237 0.625000
Model 3: Tuned VGG16 (32) - RGB 0.784367 0.712879 0.812500

Think About It:

  • What do you infer from the general trend in the training performance?
  • Is the training accuracy consistently improving?
  • Is the validation accuracy also improving similarly?

Note: You can even go back and build your own architecture on top of the VGG16 Transfer layer and see if you can improve the performance

⏩ Observations and Insights:

  • The VGG16 is a convolutional neural network model that's used for image recognition. This neural network was trained on more than a million images from the ImageNet database.
  • For the current approach in model no. 3, it was taken from VGG16 the convolutional layers block (18 layers), and joined to the classification layers composed mainly of Dense layers, the idea was to verify if this approach improves the results obtained with the previous models.
  • On the first try, it was turned the trainable property of all the convolutional layers (extracted from VGG16), to False, so it was ensured that the weight learned in VGG16 will be applied to the new problem. This provided 57.3% of accuracy, not good at all.
  • In the second try, the trainable property = False was not applied to all convolutional layers and when the model was compiled it allows the backpropagation to update the last 4 pre-trained layers. The 4 last convolutional layers were trained with the data to help the model learn specific features from the data in this specific image classification problem. This second try achieved 71.3% of accuracy. This tuning improves the performance of the model.
  • The learning curve reflects overfitting, it is necessary to explore if adding Dropout layers just after the convolutional layers could help to deal with this problem.
  • Although the result of "Model 3: Tuned VGG16 (32) - RGB" is lower than the obtained of the previous models ("Model 1: Base Neural Network (32) - RGB" and "Model 2: Enhanced Neural Network (64) - RGB"), the general performance in the training model is higher.
  • Happy and Surprise are the best-evaluated classes. Neutral and Sad are the classes with the lowest performance.
  • The best model "Model 3: Tuned VGG16 (32) - RGB" achieved 71.3% accuracy.

⏩ Recommendations:

  • Technique to deal with the overfitting needs to be applied to the model "Model 3: Tuned VGG16 (32) - RGB" and see if this improves the result in the validation set.
  • Explore other image augmentation techniques to provide more data to the model.
  • A review of the training dataset is necessary to confirm data are good enough.

Model 4: ResNet V2¶


  • Import Resnet v2 upto the layer of your choice and add Fully Connected layers on top of it.
In [55]:
ap.ResNet101(include_top=False, weights="imagenet", input_shape=(48, 48, 3)).summary()
Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/resnet/resnet101_weights_tf_dim_ordering_tf_kernels_notop.h5
171446536/171446536 [==============================] - 1s 0us/step
Model: "resnet101"
__________________________________________________________________________________________________
 Layer (type)                   Output Shape         Param #     Connected to                     
==================================================================================================
 input_2 (InputLayer)           [(None, 48, 48, 3)]  0           []                               
                                                                                                  
 conv1_pad (ZeroPadding2D)      (None, 54, 54, 3)    0           ['input_2[0][0]']                
                                                                                                  
 conv1_conv (Conv2D)            (None, 24, 24, 64)   9472        ['conv1_pad[0][0]']              
                                                                                                  
 conv1_bn (BatchNormalization)  (None, 24, 24, 64)   256         ['conv1_conv[0][0]']             
                                                                                                  
 conv1_relu (Activation)        (None, 24, 24, 64)   0           ['conv1_bn[0][0]']               
                                                                                                  
 pool1_pad (ZeroPadding2D)      (None, 26, 26, 64)   0           ['conv1_relu[0][0]']             
                                                                                                  
 pool1_pool (MaxPooling2D)      (None, 12, 12, 64)   0           ['pool1_pad[0][0]']              
                                                                                                  
 conv2_block1_1_conv (Conv2D)   (None, 12, 12, 64)   4160        ['pool1_pool[0][0]']             
                                                                                                  
 conv2_block1_1_bn (BatchNormal  (None, 12, 12, 64)  256         ['conv2_block1_1_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv2_block1_1_relu (Activatio  (None, 12, 12, 64)  0           ['conv2_block1_1_bn[0][0]']      
 n)                                                                                               
                                                                                                  
 conv2_block1_2_conv (Conv2D)   (None, 12, 12, 64)   36928       ['conv2_block1_1_relu[0][0]']    
                                                                                                  
 conv2_block1_2_bn (BatchNormal  (None, 12, 12, 64)  256         ['conv2_block1_2_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv2_block1_2_relu (Activatio  (None, 12, 12, 64)  0           ['conv2_block1_2_bn[0][0]']      
 n)                                                                                               
                                                                                                  
 conv2_block1_0_conv (Conv2D)   (None, 12, 12, 256)  16640       ['pool1_pool[0][0]']             
                                                                                                  
 conv2_block1_3_conv (Conv2D)   (None, 12, 12, 256)  16640       ['conv2_block1_2_relu[0][0]']    
                                                                                                  
 conv2_block1_0_bn (BatchNormal  (None, 12, 12, 256)  1024       ['conv2_block1_0_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv2_block1_3_bn (BatchNormal  (None, 12, 12, 256)  1024       ['conv2_block1_3_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv2_block1_add (Add)         (None, 12, 12, 256)  0           ['conv2_block1_0_bn[0][0]',      
                                                                  'conv2_block1_3_bn[0][0]']      
                                                                                                  
 conv2_block1_out (Activation)  (None, 12, 12, 256)  0           ['conv2_block1_add[0][0]']       
                                                                                                  
 conv2_block2_1_conv (Conv2D)   (None, 12, 12, 64)   16448       ['conv2_block1_out[0][0]']       
                                                                                                  
 conv2_block2_1_bn (BatchNormal  (None, 12, 12, 64)  256         ['conv2_block2_1_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv2_block2_1_relu (Activatio  (None, 12, 12, 64)  0           ['conv2_block2_1_bn[0][0]']      
 n)                                                                                               
                                                                                                  
 conv2_block2_2_conv (Conv2D)   (None, 12, 12, 64)   36928       ['conv2_block2_1_relu[0][0]']    
                                                                                                  
 conv2_block2_2_bn (BatchNormal  (None, 12, 12, 64)  256         ['conv2_block2_2_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv2_block2_2_relu (Activatio  (None, 12, 12, 64)  0           ['conv2_block2_2_bn[0][0]']      
 n)                                                                                               
                                                                                                  
 conv2_block2_3_conv (Conv2D)   (None, 12, 12, 256)  16640       ['conv2_block2_2_relu[0][0]']    
                                                                                                  
 conv2_block2_3_bn (BatchNormal  (None, 12, 12, 256)  1024       ['conv2_block2_3_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv2_block2_add (Add)         (None, 12, 12, 256)  0           ['conv2_block1_out[0][0]',       
                                                                  'conv2_block2_3_bn[0][0]']      
                                                                                                  
 conv2_block2_out (Activation)  (None, 12, 12, 256)  0           ['conv2_block2_add[0][0]']       
                                                                                                  
 conv2_block3_1_conv (Conv2D)   (None, 12, 12, 64)   16448       ['conv2_block2_out[0][0]']       
                                                                                                  
 conv2_block3_1_bn (BatchNormal  (None, 12, 12, 64)  256         ['conv2_block3_1_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv2_block3_1_relu (Activatio  (None, 12, 12, 64)  0           ['conv2_block3_1_bn[0][0]']      
 n)                                                                                               
                                                                                                  
 conv2_block3_2_conv (Conv2D)   (None, 12, 12, 64)   36928       ['conv2_block3_1_relu[0][0]']    
                                                                                                  
 conv2_block3_2_bn (BatchNormal  (None, 12, 12, 64)  256         ['conv2_block3_2_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv2_block3_2_relu (Activatio  (None, 12, 12, 64)  0           ['conv2_block3_2_bn[0][0]']      
 n)                                                                                               
                                                                                                  
 conv2_block3_3_conv (Conv2D)   (None, 12, 12, 256)  16640       ['conv2_block3_2_relu[0][0]']    
                                                                                                  
 conv2_block3_3_bn (BatchNormal  (None, 12, 12, 256)  1024       ['conv2_block3_3_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv2_block3_add (Add)         (None, 12, 12, 256)  0           ['conv2_block2_out[0][0]',       
                                                                  'conv2_block3_3_bn[0][0]']      
                                                                                                  
 conv2_block3_out (Activation)  (None, 12, 12, 256)  0           ['conv2_block3_add[0][0]']       
                                                                                                  
 conv3_block1_1_conv (Conv2D)   (None, 6, 6, 128)    32896       ['conv2_block3_out[0][0]']       
                                                                                                  
 conv3_block1_1_bn (BatchNormal  (None, 6, 6, 128)   512         ['conv3_block1_1_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv3_block1_1_relu (Activatio  (None, 6, 6, 128)   0           ['conv3_block1_1_bn[0][0]']      
 n)                                                                                               
                                                                                                  
 conv3_block1_2_conv (Conv2D)   (None, 6, 6, 128)    147584      ['conv3_block1_1_relu[0][0]']    
                                                                                                  
 conv3_block1_2_bn (BatchNormal  (None, 6, 6, 128)   512         ['conv3_block1_2_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv3_block1_2_relu (Activatio  (None, 6, 6, 128)   0           ['conv3_block1_2_bn[0][0]']      
 n)                                                                                               
                                                                                                  
 conv3_block1_0_conv (Conv2D)   (None, 6, 6, 512)    131584      ['conv2_block3_out[0][0]']       
                                                                                                  
 conv3_block1_3_conv (Conv2D)   (None, 6, 6, 512)    66048       ['conv3_block1_2_relu[0][0]']    
                                                                                                  
 conv3_block1_0_bn (BatchNormal  (None, 6, 6, 512)   2048        ['conv3_block1_0_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv3_block1_3_bn (BatchNormal  (None, 6, 6, 512)   2048        ['conv3_block1_3_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv3_block1_add (Add)         (None, 6, 6, 512)    0           ['conv3_block1_0_bn[0][0]',      
                                                                  'conv3_block1_3_bn[0][0]']      
                                                                                                  
 conv3_block1_out (Activation)  (None, 6, 6, 512)    0           ['conv3_block1_add[0][0]']       
                                                                                                  
 conv3_block2_1_conv (Conv2D)   (None, 6, 6, 128)    65664       ['conv3_block1_out[0][0]']       
                                                                                                  
 conv3_block2_1_bn (BatchNormal  (None, 6, 6, 128)   512         ['conv3_block2_1_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv3_block2_1_relu (Activatio  (None, 6, 6, 128)   0           ['conv3_block2_1_bn[0][0]']      
 n)                                                                                               
                                                                                                  
 conv3_block2_2_conv (Conv2D)   (None, 6, 6, 128)    147584      ['conv3_block2_1_relu[0][0]']    
                                                                                                  
 conv3_block2_2_bn (BatchNormal  (None, 6, 6, 128)   512         ['conv3_block2_2_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv3_block2_2_relu (Activatio  (None, 6, 6, 128)   0           ['conv3_block2_2_bn[0][0]']      
 n)                                                                                               
                                                                                                  
 conv3_block2_3_conv (Conv2D)   (None, 6, 6, 512)    66048       ['conv3_block2_2_relu[0][0]']    
                                                                                                  
 conv3_block2_3_bn (BatchNormal  (None, 6, 6, 512)   2048        ['conv3_block2_3_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv3_block2_add (Add)         (None, 6, 6, 512)    0           ['conv3_block1_out[0][0]',       
                                                                  'conv3_block2_3_bn[0][0]']      
                                                                                                  
 conv3_block2_out (Activation)  (None, 6, 6, 512)    0           ['conv3_block2_add[0][0]']       
                                                                                                  
 conv3_block3_1_conv (Conv2D)   (None, 6, 6, 128)    65664       ['conv3_block2_out[0][0]']       
                                                                                                  
 conv3_block3_1_bn (BatchNormal  (None, 6, 6, 128)   512         ['conv3_block3_1_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv3_block3_1_relu (Activatio  (None, 6, 6, 128)   0           ['conv3_block3_1_bn[0][0]']      
 n)                                                                                               
                                                                                                  
 conv3_block3_2_conv (Conv2D)   (None, 6, 6, 128)    147584      ['conv3_block3_1_relu[0][0]']    
                                                                                                  
 conv3_block3_2_bn (BatchNormal  (None, 6, 6, 128)   512         ['conv3_block3_2_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv3_block3_2_relu (Activatio  (None, 6, 6, 128)   0           ['conv3_block3_2_bn[0][0]']      
 n)                                                                                               
                                                                                                  
 conv3_block3_3_conv (Conv2D)   (None, 6, 6, 512)    66048       ['conv3_block3_2_relu[0][0]']    
                                                                                                  
 conv3_block3_3_bn (BatchNormal  (None, 6, 6, 512)   2048        ['conv3_block3_3_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv3_block3_add (Add)         (None, 6, 6, 512)    0           ['conv3_block2_out[0][0]',       
                                                                  'conv3_block3_3_bn[0][0]']      
                                                                                                  
 conv3_block3_out (Activation)  (None, 6, 6, 512)    0           ['conv3_block3_add[0][0]']       
                                                                                                  
 conv3_block4_1_conv (Conv2D)   (None, 6, 6, 128)    65664       ['conv3_block3_out[0][0]']       
                                                                                                  
 conv3_block4_1_bn (BatchNormal  (None, 6, 6, 128)   512         ['conv3_block4_1_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv3_block4_1_relu (Activatio  (None, 6, 6, 128)   0           ['conv3_block4_1_bn[0][0]']      
 n)                                                                                               
                                                                                                  
 conv3_block4_2_conv (Conv2D)   (None, 6, 6, 128)    147584      ['conv3_block4_1_relu[0][0]']    
                                                                                                  
 conv3_block4_2_bn (BatchNormal  (None, 6, 6, 128)   512         ['conv3_block4_2_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv3_block4_2_relu (Activatio  (None, 6, 6, 128)   0           ['conv3_block4_2_bn[0][0]']      
 n)                                                                                               
                                                                                                  
 conv3_block4_3_conv (Conv2D)   (None, 6, 6, 512)    66048       ['conv3_block4_2_relu[0][0]']    
                                                                                                  
 conv3_block4_3_bn (BatchNormal  (None, 6, 6, 512)   2048        ['conv3_block4_3_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv3_block4_add (Add)         (None, 6, 6, 512)    0           ['conv3_block3_out[0][0]',       
                                                                  'conv3_block4_3_bn[0][0]']      
                                                                                                  
 conv3_block4_out (Activation)  (None, 6, 6, 512)    0           ['conv3_block4_add[0][0]']       
                                                                                                  
 conv4_block1_1_conv (Conv2D)   (None, 3, 3, 256)    131328      ['conv3_block4_out[0][0]']       
                                                                                                  
 conv4_block1_1_bn (BatchNormal  (None, 3, 3, 256)   1024        ['conv4_block1_1_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv4_block1_1_relu (Activatio  (None, 3, 3, 256)   0           ['conv4_block1_1_bn[0][0]']      
 n)                                                                                               
                                                                                                  
 conv4_block1_2_conv (Conv2D)   (None, 3, 3, 256)    590080      ['conv4_block1_1_relu[0][0]']    
                                                                                                  
 conv4_block1_2_bn (BatchNormal  (None, 3, 3, 256)   1024        ['conv4_block1_2_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv4_block1_2_relu (Activatio  (None, 3, 3, 256)   0           ['conv4_block1_2_bn[0][0]']      
 n)                                                                                               
                                                                                                  
 conv4_block1_0_conv (Conv2D)   (None, 3, 3, 1024)   525312      ['conv3_block4_out[0][0]']       
                                                                                                  
 conv4_block1_3_conv (Conv2D)   (None, 3, 3, 1024)   263168      ['conv4_block1_2_relu[0][0]']    
                                                                                                  
 conv4_block1_0_bn (BatchNormal  (None, 3, 3, 1024)  4096        ['conv4_block1_0_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv4_block1_3_bn (BatchNormal  (None, 3, 3, 1024)  4096        ['conv4_block1_3_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv4_block1_add (Add)         (None, 3, 3, 1024)   0           ['conv4_block1_0_bn[0][0]',      
                                                                  'conv4_block1_3_bn[0][0]']      
                                                                                                  
 conv4_block1_out (Activation)  (None, 3, 3, 1024)   0           ['conv4_block1_add[0][0]']       
                                                                                                  
 conv4_block2_1_conv (Conv2D)   (None, 3, 3, 256)    262400      ['conv4_block1_out[0][0]']       
                                                                                                  
 conv4_block2_1_bn (BatchNormal  (None, 3, 3, 256)   1024        ['conv4_block2_1_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv4_block2_1_relu (Activatio  (None, 3, 3, 256)   0           ['conv4_block2_1_bn[0][0]']      
 n)                                                                                               
                                                                                                  
 conv4_block2_2_conv (Conv2D)   (None, 3, 3, 256)    590080      ['conv4_block2_1_relu[0][0]']    
                                                                                                  
 conv4_block2_2_bn (BatchNormal  (None, 3, 3, 256)   1024        ['conv4_block2_2_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv4_block2_2_relu (Activatio  (None, 3, 3, 256)   0           ['conv4_block2_2_bn[0][0]']      
 n)                                                                                               
                                                                                                  
 conv4_block2_3_conv (Conv2D)   (None, 3, 3, 1024)   263168      ['conv4_block2_2_relu[0][0]']    
                                                                                                  
 conv4_block2_3_bn (BatchNormal  (None, 3, 3, 1024)  4096        ['conv4_block2_3_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv4_block2_add (Add)         (None, 3, 3, 1024)   0           ['conv4_block1_out[0][0]',       
                                                                  'conv4_block2_3_bn[0][0]']      
                                                                                                  
 conv4_block2_out (Activation)  (None, 3, 3, 1024)   0           ['conv4_block2_add[0][0]']       
                                                                                                  
 conv4_block3_1_conv (Conv2D)   (None, 3, 3, 256)    262400      ['conv4_block2_out[0][0]']       
                                                                                                  
 conv4_block3_1_bn (BatchNormal  (None, 3, 3, 256)   1024        ['conv4_block3_1_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv4_block3_1_relu (Activatio  (None, 3, 3, 256)   0           ['conv4_block3_1_bn[0][0]']      
 n)                                                                                               
                                                                                                  
 conv4_block3_2_conv (Conv2D)   (None, 3, 3, 256)    590080      ['conv4_block3_1_relu[0][0]']    
                                                                                                  
 conv4_block3_2_bn (BatchNormal  (None, 3, 3, 256)   1024        ['conv4_block3_2_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv4_block3_2_relu (Activatio  (None, 3, 3, 256)   0           ['conv4_block3_2_bn[0][0]']      
 n)                                                                                               
                                                                                                  
 conv4_block3_3_conv (Conv2D)   (None, 3, 3, 1024)   263168      ['conv4_block3_2_relu[0][0]']    
                                                                                                  
 conv4_block3_3_bn (BatchNormal  (None, 3, 3, 1024)  4096        ['conv4_block3_3_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv4_block3_add (Add)         (None, 3, 3, 1024)   0           ['conv4_block2_out[0][0]',       
                                                                  'conv4_block3_3_bn[0][0]']      
                                                                                                  
 conv4_block3_out (Activation)  (None, 3, 3, 1024)   0           ['conv4_block3_add[0][0]']       
                                                                                                  
 conv4_block4_1_conv (Conv2D)   (None, 3, 3, 256)    262400      ['conv4_block3_out[0][0]']       
                                                                                                  
 conv4_block4_1_bn (BatchNormal  (None, 3, 3, 256)   1024        ['conv4_block4_1_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv4_block4_1_relu (Activatio  (None, 3, 3, 256)   0           ['conv4_block4_1_bn[0][0]']      
 n)                                                                                               
                                                                                                  
 conv4_block4_2_conv (Conv2D)   (None, 3, 3, 256)    590080      ['conv4_block4_1_relu[0][0]']    
                                                                                                  
 conv4_block4_2_bn (BatchNormal  (None, 3, 3, 256)   1024        ['conv4_block4_2_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv4_block4_2_relu (Activatio  (None, 3, 3, 256)   0           ['conv4_block4_2_bn[0][0]']      
 n)                                                                                               
                                                                                                  
 conv4_block4_3_conv (Conv2D)   (None, 3, 3, 1024)   263168      ['conv4_block4_2_relu[0][0]']    
                                                                                                  
 conv4_block4_3_bn (BatchNormal  (None, 3, 3, 1024)  4096        ['conv4_block4_3_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv4_block4_add (Add)         (None, 3, 3, 1024)   0           ['conv4_block3_out[0][0]',       
                                                                  'conv4_block4_3_bn[0][0]']      
                                                                                                  
 conv4_block4_out (Activation)  (None, 3, 3, 1024)   0           ['conv4_block4_add[0][0]']       
                                                                                                  
 conv4_block5_1_conv (Conv2D)   (None, 3, 3, 256)    262400      ['conv4_block4_out[0][0]']       
                                                                                                  
 conv4_block5_1_bn (BatchNormal  (None, 3, 3, 256)   1024        ['conv4_block5_1_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv4_block5_1_relu (Activatio  (None, 3, 3, 256)   0           ['conv4_block5_1_bn[0][0]']      
 n)                                                                                               
                                                                                                  
 conv4_block5_2_conv (Conv2D)   (None, 3, 3, 256)    590080      ['conv4_block5_1_relu[0][0]']    
                                                                                                  
 conv4_block5_2_bn (BatchNormal  (None, 3, 3, 256)   1024        ['conv4_block5_2_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv4_block5_2_relu (Activatio  (None, 3, 3, 256)   0           ['conv4_block5_2_bn[0][0]']      
 n)                                                                                               
                                                                                                  
 conv4_block5_3_conv (Conv2D)   (None, 3, 3, 1024)   263168      ['conv4_block5_2_relu[0][0]']    
                                                                                                  
 conv4_block5_3_bn (BatchNormal  (None, 3, 3, 1024)  4096        ['conv4_block5_3_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv4_block5_add (Add)         (None, 3, 3, 1024)   0           ['conv4_block4_out[0][0]',       
                                                                  'conv4_block5_3_bn[0][0]']      
                                                                                                  
 conv4_block5_out (Activation)  (None, 3, 3, 1024)   0           ['conv4_block5_add[0][0]']       
                                                                                                  
 conv4_block6_1_conv (Conv2D)   (None, 3, 3, 256)    262400      ['conv4_block5_out[0][0]']       
                                                                                                  
 conv4_block6_1_bn (BatchNormal  (None, 3, 3, 256)   1024        ['conv4_block6_1_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv4_block6_1_relu (Activatio  (None, 3, 3, 256)   0           ['conv4_block6_1_bn[0][0]']      
 n)                                                                                               
                                                                                                  
 conv4_block6_2_conv (Conv2D)   (None, 3, 3, 256)    590080      ['conv4_block6_1_relu[0][0]']    
                                                                                                  
 conv4_block6_2_bn (BatchNormal  (None, 3, 3, 256)   1024        ['conv4_block6_2_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv4_block6_2_relu (Activatio  (None, 3, 3, 256)   0           ['conv4_block6_2_bn[0][0]']      
 n)                                                                                               
                                                                                                  
 conv4_block6_3_conv (Conv2D)   (None, 3, 3, 1024)   263168      ['conv4_block6_2_relu[0][0]']    
                                                                                                  
 conv4_block6_3_bn (BatchNormal  (None, 3, 3, 1024)  4096        ['conv4_block6_3_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv4_block6_add (Add)         (None, 3, 3, 1024)   0           ['conv4_block5_out[0][0]',       
                                                                  'conv4_block6_3_bn[0][0]']      
                                                                                                  
 conv4_block6_out (Activation)  (None, 3, 3, 1024)   0           ['conv4_block6_add[0][0]']       
                                                                                                  
 conv4_block7_1_conv (Conv2D)   (None, 3, 3, 256)    262400      ['conv4_block6_out[0][0]']       
                                                                                                  
 conv4_block7_1_bn (BatchNormal  (None, 3, 3, 256)   1024        ['conv4_block7_1_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv4_block7_1_relu (Activatio  (None, 3, 3, 256)   0           ['conv4_block7_1_bn[0][0]']      
 n)                                                                                               
                                                                                                  
 conv4_block7_2_conv (Conv2D)   (None, 3, 3, 256)    590080      ['conv4_block7_1_relu[0][0]']    
                                                                                                  
 conv4_block7_2_bn (BatchNormal  (None, 3, 3, 256)   1024        ['conv4_block7_2_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv4_block7_2_relu (Activatio  (None, 3, 3, 256)   0           ['conv4_block7_2_bn[0][0]']      
 n)                                                                                               
                                                                                                  
 conv4_block7_3_conv (Conv2D)   (None, 3, 3, 1024)   263168      ['conv4_block7_2_relu[0][0]']    
                                                                                                  
 conv4_block7_3_bn (BatchNormal  (None, 3, 3, 1024)  4096        ['conv4_block7_3_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv4_block7_add (Add)         (None, 3, 3, 1024)   0           ['conv4_block6_out[0][0]',       
                                                                  'conv4_block7_3_bn[0][0]']      
                                                                                                  
 conv4_block7_out (Activation)  (None, 3, 3, 1024)   0           ['conv4_block7_add[0][0]']       
                                                                                                  
 conv4_block8_1_conv (Conv2D)   (None, 3, 3, 256)    262400      ['conv4_block7_out[0][0]']       
                                                                                                  
 conv4_block8_1_bn (BatchNormal  (None, 3, 3, 256)   1024        ['conv4_block8_1_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv4_block8_1_relu (Activatio  (None, 3, 3, 256)   0           ['conv4_block8_1_bn[0][0]']      
 n)                                                                                               
                                                                                                  
 conv4_block8_2_conv (Conv2D)   (None, 3, 3, 256)    590080      ['conv4_block8_1_relu[0][0]']    
                                                                                                  
 conv4_block8_2_bn (BatchNormal  (None, 3, 3, 256)   1024        ['conv4_block8_2_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv4_block8_2_relu (Activatio  (None, 3, 3, 256)   0           ['conv4_block8_2_bn[0][0]']      
 n)                                                                                               
                                                                                                  
 conv4_block8_3_conv (Conv2D)   (None, 3, 3, 1024)   263168      ['conv4_block8_2_relu[0][0]']    
                                                                                                  
 conv4_block8_3_bn (BatchNormal  (None, 3, 3, 1024)  4096        ['conv4_block8_3_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv4_block8_add (Add)         (None, 3, 3, 1024)   0           ['conv4_block7_out[0][0]',       
                                                                  'conv4_block8_3_bn[0][0]']      
                                                                                                  
 conv4_block8_out (Activation)  (None, 3, 3, 1024)   0           ['conv4_block8_add[0][0]']       
                                                                                                  
 conv4_block9_1_conv (Conv2D)   (None, 3, 3, 256)    262400      ['conv4_block8_out[0][0]']       
                                                                                                  
 conv4_block9_1_bn (BatchNormal  (None, 3, 3, 256)   1024        ['conv4_block9_1_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv4_block9_1_relu (Activatio  (None, 3, 3, 256)   0           ['conv4_block9_1_bn[0][0]']      
 n)                                                                                               
                                                                                                  
 conv4_block9_2_conv (Conv2D)   (None, 3, 3, 256)    590080      ['conv4_block9_1_relu[0][0]']    
                                                                                                  
 conv4_block9_2_bn (BatchNormal  (None, 3, 3, 256)   1024        ['conv4_block9_2_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv4_block9_2_relu (Activatio  (None, 3, 3, 256)   0           ['conv4_block9_2_bn[0][0]']      
 n)                                                                                               
                                                                                                  
 conv4_block9_3_conv (Conv2D)   (None, 3, 3, 1024)   263168      ['conv4_block9_2_relu[0][0]']    
                                                                                                  
 conv4_block9_3_bn (BatchNormal  (None, 3, 3, 1024)  4096        ['conv4_block9_3_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv4_block9_add (Add)         (None, 3, 3, 1024)   0           ['conv4_block8_out[0][0]',       
                                                                  'conv4_block9_3_bn[0][0]']      
                                                                                                  
 conv4_block9_out (Activation)  (None, 3, 3, 1024)   0           ['conv4_block9_add[0][0]']       
                                                                                                  
 conv4_block10_1_conv (Conv2D)  (None, 3, 3, 256)    262400      ['conv4_block9_out[0][0]']       
                                                                                                  
 conv4_block10_1_bn (BatchNorma  (None, 3, 3, 256)   1024        ['conv4_block10_1_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 conv4_block10_1_relu (Activati  (None, 3, 3, 256)   0           ['conv4_block10_1_bn[0][0]']     
 on)                                                                                              
                                                                                                  
 conv4_block10_2_conv (Conv2D)  (None, 3, 3, 256)    590080      ['conv4_block10_1_relu[0][0]']   
                                                                                                  
 conv4_block10_2_bn (BatchNorma  (None, 3, 3, 256)   1024        ['conv4_block10_2_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 conv4_block10_2_relu (Activati  (None, 3, 3, 256)   0           ['conv4_block10_2_bn[0][0]']     
 on)                                                                                              
                                                                                                  
 conv4_block10_3_conv (Conv2D)  (None, 3, 3, 1024)   263168      ['conv4_block10_2_relu[0][0]']   
                                                                                                  
 conv4_block10_3_bn (BatchNorma  (None, 3, 3, 1024)  4096        ['conv4_block10_3_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 conv4_block10_add (Add)        (None, 3, 3, 1024)   0           ['conv4_block9_out[0][0]',       
                                                                  'conv4_block10_3_bn[0][0]']     
                                                                                                  
 conv4_block10_out (Activation)  (None, 3, 3, 1024)  0           ['conv4_block10_add[0][0]']      
                                                                                                  
 conv4_block11_1_conv (Conv2D)  (None, 3, 3, 256)    262400      ['conv4_block10_out[0][0]']      
                                                                                                  
 conv4_block11_1_bn (BatchNorma  (None, 3, 3, 256)   1024        ['conv4_block11_1_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 conv4_block11_1_relu (Activati  (None, 3, 3, 256)   0           ['conv4_block11_1_bn[0][0]']     
 on)                                                                                              
                                                                                                  
 conv4_block11_2_conv (Conv2D)  (None, 3, 3, 256)    590080      ['conv4_block11_1_relu[0][0]']   
                                                                                                  
 conv4_block11_2_bn (BatchNorma  (None, 3, 3, 256)   1024        ['conv4_block11_2_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 conv4_block11_2_relu (Activati  (None, 3, 3, 256)   0           ['conv4_block11_2_bn[0][0]']     
 on)                                                                                              
                                                                                                  
 conv4_block11_3_conv (Conv2D)  (None, 3, 3, 1024)   263168      ['conv4_block11_2_relu[0][0]']   
                                                                                                  
 conv4_block11_3_bn (BatchNorma  (None, 3, 3, 1024)  4096        ['conv4_block11_3_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 conv4_block11_add (Add)        (None, 3, 3, 1024)   0           ['conv4_block10_out[0][0]',      
                                                                  'conv4_block11_3_bn[0][0]']     
                                                                                                  
 conv4_block11_out (Activation)  (None, 3, 3, 1024)  0           ['conv4_block11_add[0][0]']      
                                                                                                  
 conv4_block12_1_conv (Conv2D)  (None, 3, 3, 256)    262400      ['conv4_block11_out[0][0]']      
                                                                                                  
 conv4_block12_1_bn (BatchNorma  (None, 3, 3, 256)   1024        ['conv4_block12_1_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 conv4_block12_1_relu (Activati  (None, 3, 3, 256)   0           ['conv4_block12_1_bn[0][0]']     
 on)                                                                                              
                                                                                                  
 conv4_block12_2_conv (Conv2D)  (None, 3, 3, 256)    590080      ['conv4_block12_1_relu[0][0]']   
                                                                                                  
 conv4_block12_2_bn (BatchNorma  (None, 3, 3, 256)   1024        ['conv4_block12_2_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 conv4_block12_2_relu (Activati  (None, 3, 3, 256)   0           ['conv4_block12_2_bn[0][0]']     
 on)                                                                                              
                                                                                                  
 conv4_block12_3_conv (Conv2D)  (None, 3, 3, 1024)   263168      ['conv4_block12_2_relu[0][0]']   
                                                                                                  
 conv4_block12_3_bn (BatchNorma  (None, 3, 3, 1024)  4096        ['conv4_block12_3_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 conv4_block12_add (Add)        (None, 3, 3, 1024)   0           ['conv4_block11_out[0][0]',      
                                                                  'conv4_block12_3_bn[0][0]']     
                                                                                                  
 conv4_block12_out (Activation)  (None, 3, 3, 1024)  0           ['conv4_block12_add[0][0]']      
                                                                                                  
 conv4_block13_1_conv (Conv2D)  (None, 3, 3, 256)    262400      ['conv4_block12_out[0][0]']      
                                                                                                  
 conv4_block13_1_bn (BatchNorma  (None, 3, 3, 256)   1024        ['conv4_block13_1_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 conv4_block13_1_relu (Activati  (None, 3, 3, 256)   0           ['conv4_block13_1_bn[0][0]']     
 on)                                                                                              
                                                                                                  
 conv4_block13_2_conv (Conv2D)  (None, 3, 3, 256)    590080      ['conv4_block13_1_relu[0][0]']   
                                                                                                  
 conv4_block13_2_bn (BatchNorma  (None, 3, 3, 256)   1024        ['conv4_block13_2_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 conv4_block13_2_relu (Activati  (None, 3, 3, 256)   0           ['conv4_block13_2_bn[0][0]']     
 on)                                                                                              
                                                                                                  
 conv4_block13_3_conv (Conv2D)  (None, 3, 3, 1024)   263168      ['conv4_block13_2_relu[0][0]']   
                                                                                                  
 conv4_block13_3_bn (BatchNorma  (None, 3, 3, 1024)  4096        ['conv4_block13_3_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 conv4_block13_add (Add)        (None, 3, 3, 1024)   0           ['conv4_block12_out[0][0]',      
                                                                  'conv4_block13_3_bn[0][0]']     
                                                                                                  
 conv4_block13_out (Activation)  (None, 3, 3, 1024)  0           ['conv4_block13_add[0][0]']      
                                                                                                  
 conv4_block14_1_conv (Conv2D)  (None, 3, 3, 256)    262400      ['conv4_block13_out[0][0]']      
                                                                                                  
 conv4_block14_1_bn (BatchNorma  (None, 3, 3, 256)   1024        ['conv4_block14_1_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 conv4_block14_1_relu (Activati  (None, 3, 3, 256)   0           ['conv4_block14_1_bn[0][0]']     
 on)                                                                                              
                                                                                                  
 conv4_block14_2_conv (Conv2D)  (None, 3, 3, 256)    590080      ['conv4_block14_1_relu[0][0]']   
                                                                                                  
 conv4_block14_2_bn (BatchNorma  (None, 3, 3, 256)   1024        ['conv4_block14_2_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 conv4_block14_2_relu (Activati  (None, 3, 3, 256)   0           ['conv4_block14_2_bn[0][0]']     
 on)                                                                                              
                                                                                                  
 conv4_block14_3_conv (Conv2D)  (None, 3, 3, 1024)   263168      ['conv4_block14_2_relu[0][0]']   
                                                                                                  
 conv4_block14_3_bn (BatchNorma  (None, 3, 3, 1024)  4096        ['conv4_block14_3_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 conv4_block14_add (Add)        (None, 3, 3, 1024)   0           ['conv4_block13_out[0][0]',      
                                                                  'conv4_block14_3_bn[0][0]']     
                                                                                                  
 conv4_block14_out (Activation)  (None, 3, 3, 1024)  0           ['conv4_block14_add[0][0]']      
                                                                                                  
 conv4_block15_1_conv (Conv2D)  (None, 3, 3, 256)    262400      ['conv4_block14_out[0][0]']      
                                                                                                  
 conv4_block15_1_bn (BatchNorma  (None, 3, 3, 256)   1024        ['conv4_block15_1_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 conv4_block15_1_relu (Activati  (None, 3, 3, 256)   0           ['conv4_block15_1_bn[0][0]']     
 on)                                                                                              
                                                                                                  
 conv4_block15_2_conv (Conv2D)  (None, 3, 3, 256)    590080      ['conv4_block15_1_relu[0][0]']   
                                                                                                  
 conv4_block15_2_bn (BatchNorma  (None, 3, 3, 256)   1024        ['conv4_block15_2_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 conv4_block15_2_relu (Activati  (None, 3, 3, 256)   0           ['conv4_block15_2_bn[0][0]']     
 on)                                                                                              
                                                                                                  
 conv4_block15_3_conv (Conv2D)  (None, 3, 3, 1024)   263168      ['conv4_block15_2_relu[0][0]']   
                                                                                                  
 conv4_block15_3_bn (BatchNorma  (None, 3, 3, 1024)  4096        ['conv4_block15_3_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 conv4_block15_add (Add)        (None, 3, 3, 1024)   0           ['conv4_block14_out[0][0]',      
                                                                  'conv4_block15_3_bn[0][0]']     
                                                                                                  
 conv4_block15_out (Activation)  (None, 3, 3, 1024)  0           ['conv4_block15_add[0][0]']      
                                                                                                  
 conv4_block16_1_conv (Conv2D)  (None, 3, 3, 256)    262400      ['conv4_block15_out[0][0]']      
                                                                                                  
 conv4_block16_1_bn (BatchNorma  (None, 3, 3, 256)   1024        ['conv4_block16_1_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 conv4_block16_1_relu (Activati  (None, 3, 3, 256)   0           ['conv4_block16_1_bn[0][0]']     
 on)                                                                                              
                                                                                                  
 conv4_block16_2_conv (Conv2D)  (None, 3, 3, 256)    590080      ['conv4_block16_1_relu[0][0]']   
                                                                                                  
 conv4_block16_2_bn (BatchNorma  (None, 3, 3, 256)   1024        ['conv4_block16_2_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 conv4_block16_2_relu (Activati  (None, 3, 3, 256)   0           ['conv4_block16_2_bn[0][0]']     
 on)                                                                                              
                                                                                                  
 conv4_block16_3_conv (Conv2D)  (None, 3, 3, 1024)   263168      ['conv4_block16_2_relu[0][0]']   
                                                                                                  
 conv4_block16_3_bn (BatchNorma  (None, 3, 3, 1024)  4096        ['conv4_block16_3_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 conv4_block16_add (Add)        (None, 3, 3, 1024)   0           ['conv4_block15_out[0][0]',      
                                                                  'conv4_block16_3_bn[0][0]']     
                                                                                                  
 conv4_block16_out (Activation)  (None, 3, 3, 1024)  0           ['conv4_block16_add[0][0]']      
                                                                                                  
 conv4_block17_1_conv (Conv2D)  (None, 3, 3, 256)    262400      ['conv4_block16_out[0][0]']      
                                                                                                  
 conv4_block17_1_bn (BatchNorma  (None, 3, 3, 256)   1024        ['conv4_block17_1_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 conv4_block17_1_relu (Activati  (None, 3, 3, 256)   0           ['conv4_block17_1_bn[0][0]']     
 on)                                                                                              
                                                                                                  
 conv4_block17_2_conv (Conv2D)  (None, 3, 3, 256)    590080      ['conv4_block17_1_relu[0][0]']   
                                                                                                  
 conv4_block17_2_bn (BatchNorma  (None, 3, 3, 256)   1024        ['conv4_block17_2_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 conv4_block17_2_relu (Activati  (None, 3, 3, 256)   0           ['conv4_block17_2_bn[0][0]']     
 on)                                                                                              
                                                                                                  
 conv4_block17_3_conv (Conv2D)  (None, 3, 3, 1024)   263168      ['conv4_block17_2_relu[0][0]']   
                                                                                                  
 conv4_block17_3_bn (BatchNorma  (None, 3, 3, 1024)  4096        ['conv4_block17_3_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 conv4_block17_add (Add)        (None, 3, 3, 1024)   0           ['conv4_block16_out[0][0]',      
                                                                  'conv4_block17_3_bn[0][0]']     
                                                                                                  
 conv4_block17_out (Activation)  (None, 3, 3, 1024)  0           ['conv4_block17_add[0][0]']      
                                                                                                  
 conv4_block18_1_conv (Conv2D)  (None, 3, 3, 256)    262400      ['conv4_block17_out[0][0]']      
                                                                                                  
 conv4_block18_1_bn (BatchNorma  (None, 3, 3, 256)   1024        ['conv4_block18_1_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 conv4_block18_1_relu (Activati  (None, 3, 3, 256)   0           ['conv4_block18_1_bn[0][0]']     
 on)                                                                                              
                                                                                                  
 conv4_block18_2_conv (Conv2D)  (None, 3, 3, 256)    590080      ['conv4_block18_1_relu[0][0]']   
                                                                                                  
 conv4_block18_2_bn (BatchNorma  (None, 3, 3, 256)   1024        ['conv4_block18_2_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 conv4_block18_2_relu (Activati  (None, 3, 3, 256)   0           ['conv4_block18_2_bn[0][0]']     
 on)                                                                                              
                                                                                                  
 conv4_block18_3_conv (Conv2D)  (None, 3, 3, 1024)   263168      ['conv4_block18_2_relu[0][0]']   
                                                                                                  
 conv4_block18_3_bn (BatchNorma  (None, 3, 3, 1024)  4096        ['conv4_block18_3_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 conv4_block18_add (Add)        (None, 3, 3, 1024)   0           ['conv4_block17_out[0][0]',      
                                                                  'conv4_block18_3_bn[0][0]']     
                                                                                                  
 conv4_block18_out (Activation)  (None, 3, 3, 1024)  0           ['conv4_block18_add[0][0]']      
                                                                                                  
 conv4_block19_1_conv (Conv2D)  (None, 3, 3, 256)    262400      ['conv4_block18_out[0][0]']      
                                                                                                  
 conv4_block19_1_bn (BatchNorma  (None, 3, 3, 256)   1024        ['conv4_block19_1_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 conv4_block19_1_relu (Activati  (None, 3, 3, 256)   0           ['conv4_block19_1_bn[0][0]']     
 on)                                                                                              
                                                                                                  
 conv4_block19_2_conv (Conv2D)  (None, 3, 3, 256)    590080      ['conv4_block19_1_relu[0][0]']   
                                                                                                  
 conv4_block19_2_bn (BatchNorma  (None, 3, 3, 256)   1024        ['conv4_block19_2_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 conv4_block19_2_relu (Activati  (None, 3, 3, 256)   0           ['conv4_block19_2_bn[0][0]']     
 on)                                                                                              
                                                                                                  
 conv4_block19_3_conv (Conv2D)  (None, 3, 3, 1024)   263168      ['conv4_block19_2_relu[0][0]']   
                                                                                                  
 conv4_block19_3_bn (BatchNorma  (None, 3, 3, 1024)  4096        ['conv4_block19_3_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 conv4_block19_add (Add)        (None, 3, 3, 1024)   0           ['conv4_block18_out[0][0]',      
                                                                  'conv4_block19_3_bn[0][0]']     
                                                                                                  
 conv4_block19_out (Activation)  (None, 3, 3, 1024)  0           ['conv4_block19_add[0][0]']      
                                                                                                  
 conv4_block20_1_conv (Conv2D)  (None, 3, 3, 256)    262400      ['conv4_block19_out[0][0]']      
                                                                                                  
 conv4_block20_1_bn (BatchNorma  (None, 3, 3, 256)   1024        ['conv4_block20_1_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 conv4_block20_1_relu (Activati  (None, 3, 3, 256)   0           ['conv4_block20_1_bn[0][0]']     
 on)                                                                                              
                                                                                                  
 conv4_block20_2_conv (Conv2D)  (None, 3, 3, 256)    590080      ['conv4_block20_1_relu[0][0]']   
                                                                                                  
 conv4_block20_2_bn (BatchNorma  (None, 3, 3, 256)   1024        ['conv4_block20_2_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 conv4_block20_2_relu (Activati  (None, 3, 3, 256)   0           ['conv4_block20_2_bn[0][0]']     
 on)                                                                                              
                                                                                                  
 conv4_block20_3_conv (Conv2D)  (None, 3, 3, 1024)   263168      ['conv4_block20_2_relu[0][0]']   
                                                                                                  
 conv4_block20_3_bn (BatchNorma  (None, 3, 3, 1024)  4096        ['conv4_block20_3_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 conv4_block20_add (Add)        (None, 3, 3, 1024)   0           ['conv4_block19_out[0][0]',      
                                                                  'conv4_block20_3_bn[0][0]']     
                                                                                                  
 conv4_block20_out (Activation)  (None, 3, 3, 1024)  0           ['conv4_block20_add[0][0]']      
                                                                                                  
 conv4_block21_1_conv (Conv2D)  (None, 3, 3, 256)    262400      ['conv4_block20_out[0][0]']      
                                                                                                  
 conv4_block21_1_bn (BatchNorma  (None, 3, 3, 256)   1024        ['conv4_block21_1_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 conv4_block21_1_relu (Activati  (None, 3, 3, 256)   0           ['conv4_block21_1_bn[0][0]']     
 on)                                                                                              
                                                                                                  
 conv4_block21_2_conv (Conv2D)  (None, 3, 3, 256)    590080      ['conv4_block21_1_relu[0][0]']   
                                                                                                  
 conv4_block21_2_bn (BatchNorma  (None, 3, 3, 256)   1024        ['conv4_block21_2_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 conv4_block21_2_relu (Activati  (None, 3, 3, 256)   0           ['conv4_block21_2_bn[0][0]']     
 on)                                                                                              
                                                                                                  
 conv4_block21_3_conv (Conv2D)  (None, 3, 3, 1024)   263168      ['conv4_block21_2_relu[0][0]']   
                                                                                                  
 conv4_block21_3_bn (BatchNorma  (None, 3, 3, 1024)  4096        ['conv4_block21_3_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 conv4_block21_add (Add)        (None, 3, 3, 1024)   0           ['conv4_block20_out[0][0]',      
                                                                  'conv4_block21_3_bn[0][0]']     
                                                                                                  
 conv4_block21_out (Activation)  (None, 3, 3, 1024)  0           ['conv4_block21_add[0][0]']      
                                                                                                  
 conv4_block22_1_conv (Conv2D)  (None, 3, 3, 256)    262400      ['conv4_block21_out[0][0]']      
                                                                                                  
 conv4_block22_1_bn (BatchNorma  (None, 3, 3, 256)   1024        ['conv4_block22_1_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 conv4_block22_1_relu (Activati  (None, 3, 3, 256)   0           ['conv4_block22_1_bn[0][0]']     
 on)                                                                                              
                                                                                                  
 conv4_block22_2_conv (Conv2D)  (None, 3, 3, 256)    590080      ['conv4_block22_1_relu[0][0]']   
                                                                                                  
 conv4_block22_2_bn (BatchNorma  (None, 3, 3, 256)   1024        ['conv4_block22_2_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 conv4_block22_2_relu (Activati  (None, 3, 3, 256)   0           ['conv4_block22_2_bn[0][0]']     
 on)                                                                                              
                                                                                                  
 conv4_block22_3_conv (Conv2D)  (None, 3, 3, 1024)   263168      ['conv4_block22_2_relu[0][0]']   
                                                                                                  
 conv4_block22_3_bn (BatchNorma  (None, 3, 3, 1024)  4096        ['conv4_block22_3_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 conv4_block22_add (Add)        (None, 3, 3, 1024)   0           ['conv4_block21_out[0][0]',      
                                                                  'conv4_block22_3_bn[0][0]']     
                                                                                                  
 conv4_block22_out (Activation)  (None, 3, 3, 1024)  0           ['conv4_block22_add[0][0]']      
                                                                                                  
 conv4_block23_1_conv (Conv2D)  (None, 3, 3, 256)    262400      ['conv4_block22_out[0][0]']      
                                                                                                  
 conv4_block23_1_bn (BatchNorma  (None, 3, 3, 256)   1024        ['conv4_block23_1_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 conv4_block23_1_relu (Activati  (None, 3, 3, 256)   0           ['conv4_block23_1_bn[0][0]']     
 on)                                                                                              
                                                                                                  
 conv4_block23_2_conv (Conv2D)  (None, 3, 3, 256)    590080      ['conv4_block23_1_relu[0][0]']   
                                                                                                  
 conv4_block23_2_bn (BatchNorma  (None, 3, 3, 256)   1024        ['conv4_block23_2_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 conv4_block23_2_relu (Activati  (None, 3, 3, 256)   0           ['conv4_block23_2_bn[0][0]']     
 on)                                                                                              
                                                                                                  
 conv4_block23_3_conv (Conv2D)  (None, 3, 3, 1024)   263168      ['conv4_block23_2_relu[0][0]']   
                                                                                                  
 conv4_block23_3_bn (BatchNorma  (None, 3, 3, 1024)  4096        ['conv4_block23_3_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 conv4_block23_add (Add)        (None, 3, 3, 1024)   0           ['conv4_block22_out[0][0]',      
                                                                  'conv4_block23_3_bn[0][0]']     
                                                                                                  
 conv4_block23_out (Activation)  (None, 3, 3, 1024)  0           ['conv4_block23_add[0][0]']      
                                                                                                  
 conv5_block1_1_conv (Conv2D)   (None, 2, 2, 512)    524800      ['conv4_block23_out[0][0]']      
                                                                                                  
 conv5_block1_1_bn (BatchNormal  (None, 2, 2, 512)   2048        ['conv5_block1_1_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv5_block1_1_relu (Activatio  (None, 2, 2, 512)   0           ['conv5_block1_1_bn[0][0]']      
 n)                                                                                               
                                                                                                  
 conv5_block1_2_conv (Conv2D)   (None, 2, 2, 512)    2359808     ['conv5_block1_1_relu[0][0]']    
                                                                                                  
 conv5_block1_2_bn (BatchNormal  (None, 2, 2, 512)   2048        ['conv5_block1_2_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv5_block1_2_relu (Activatio  (None, 2, 2, 512)   0           ['conv5_block1_2_bn[0][0]']      
 n)                                                                                               
                                                                                                  
 conv5_block1_0_conv (Conv2D)   (None, 2, 2, 2048)   2099200     ['conv4_block23_out[0][0]']      
                                                                                                  
 conv5_block1_3_conv (Conv2D)   (None, 2, 2, 2048)   1050624     ['conv5_block1_2_relu[0][0]']    
                                                                                                  
 conv5_block1_0_bn (BatchNormal  (None, 2, 2, 2048)  8192        ['conv5_block1_0_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv5_block1_3_bn (BatchNormal  (None, 2, 2, 2048)  8192        ['conv5_block1_3_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv5_block1_add (Add)         (None, 2, 2, 2048)   0           ['conv5_block1_0_bn[0][0]',      
                                                                  'conv5_block1_3_bn[0][0]']      
                                                                                                  
 conv5_block1_out (Activation)  (None, 2, 2, 2048)   0           ['conv5_block1_add[0][0]']       
                                                                                                  
 conv5_block2_1_conv (Conv2D)   (None, 2, 2, 512)    1049088     ['conv5_block1_out[0][0]']       
                                                                                                  
 conv5_block2_1_bn (BatchNormal  (None, 2, 2, 512)   2048        ['conv5_block2_1_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv5_block2_1_relu (Activatio  (None, 2, 2, 512)   0           ['conv5_block2_1_bn[0][0]']      
 n)                                                                                               
                                                                                                  
 conv5_block2_2_conv (Conv2D)   (None, 2, 2, 512)    2359808     ['conv5_block2_1_relu[0][0]']    
                                                                                                  
 conv5_block2_2_bn (BatchNormal  (None, 2, 2, 512)   2048        ['conv5_block2_2_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv5_block2_2_relu (Activatio  (None, 2, 2, 512)   0           ['conv5_block2_2_bn[0][0]']      
 n)                                                                                               
                                                                                                  
 conv5_block2_3_conv (Conv2D)   (None, 2, 2, 2048)   1050624     ['conv5_block2_2_relu[0][0]']    
                                                                                                  
 conv5_block2_3_bn (BatchNormal  (None, 2, 2, 2048)  8192        ['conv5_block2_3_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv5_block2_add (Add)         (None, 2, 2, 2048)   0           ['conv5_block1_out[0][0]',       
                                                                  'conv5_block2_3_bn[0][0]']      
                                                                                                  
 conv5_block2_out (Activation)  (None, 2, 2, 2048)   0           ['conv5_block2_add[0][0]']       
                                                                                                  
 conv5_block3_1_conv (Conv2D)   (None, 2, 2, 512)    1049088     ['conv5_block2_out[0][0]']       
                                                                                                  
 conv5_block3_1_bn (BatchNormal  (None, 2, 2, 512)   2048        ['conv5_block3_1_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv5_block3_1_relu (Activatio  (None, 2, 2, 512)   0           ['conv5_block3_1_bn[0][0]']      
 n)                                                                                               
                                                                                                  
 conv5_block3_2_conv (Conv2D)   (None, 2, 2, 512)    2359808     ['conv5_block3_1_relu[0][0]']    
                                                                                                  
 conv5_block3_2_bn (BatchNormal  (None, 2, 2, 512)   2048        ['conv5_block3_2_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv5_block3_2_relu (Activatio  (None, 2, 2, 512)   0           ['conv5_block3_2_bn[0][0]']      
 n)                                                                                               
                                                                                                  
 conv5_block3_3_conv (Conv2D)   (None, 2, 2, 2048)   1050624     ['conv5_block3_2_relu[0][0]']    
                                                                                                  
 conv5_block3_3_bn (BatchNormal  (None, 2, 2, 2048)  8192        ['conv5_block3_3_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 conv5_block3_add (Add)         (None, 2, 2, 2048)   0           ['conv5_block2_out[0][0]',       
                                                                  'conv5_block3_3_bn[0][0]']      
                                                                                                  
 conv5_block3_out (Activation)  (None, 2, 2, 2048)   0           ['conv5_block3_add[0][0]']       
                                                                                                  
==================================================================================================
Total params: 42,658,176
Trainable params: 42,552,832
Non-trainable params: 105,344
__________________________________________________________________________________________________
In [56]:
# Model 4: ResNet V2
def model_builder_resnet(input_shape, tune=0):
  """Transfer approach with the ResNet V2
  """
  resnet_model = ap.ResNet101(include_top=False, weights="imagenet", input_shape=input_shape)
  # Making all the layers of the VGG model non-trainable
  if tune > 0:
    for layer in resnet_model.layers[:-tune]:
        layer.trainable = False
  else:
    for layer in resnet_model.layers:
        layer.trainable = False

  model = Sequential([
      resnet_model,
      Flatten(),
      Dense(256, activation='relu'),
      Dense(128, activation='relu'),
      Dropout(0.3),
      Dense(64, activation='relu'),
      BatchNormalization(),
      Dense(4, activation='softmax')
  ])

  # Compile model
  optimizer = Adam(learning_rate = 0.001)
  model.compile(optimizer=optimizer, loss='categorical_crossentropy', metrics=['accuracy'])

  # Generating the summary of the model
  print(model.summary())
  return model
In [57]:
batch_size = 32
model_resnet, metric_resnet, history_resnet = model_building_and_evaluating_process(
    model_builder_resnet, f'Model 4: Resnet V2 ({batch_size})', batch_size=batch_size, epochs=20, include_grayscale=False
)
---------------------------------------------
Model 4: Resnet V2 (32) - RGB
---------------------------------------------
Initializing TF Session and Seed

Data with color_mode=rgb:
Found 15109 images belonging to 4 classes.
Found 127 images belonging to 4 classes.
Found 4977 images belonging to 4 classes.



Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 resnet101 (Functional)      (None, 2, 2, 2048)        42658176  
                                                                 
 flatten (Flatten)           (None, 8192)              0         
                                                                 
 dense (Dense)               (None, 256)               2097408   
                                                                 
 dense_1 (Dense)             (None, 128)               32896     
                                                                 
 dropout (Dropout)           (None, 128)               0         
                                                                 
 dense_2 (Dense)             (None, 64)                8256      
                                                                 
 batch_normalization (BatchN  (None, 64)               256       
 ormalization)                                                   
                                                                 
 dense_3 (Dense)             (None, 4)                 260       
                                                                 
=================================================================
Total params: 44,797,252
Trainable params: 2,138,948
Non-trainable params: 42,658,304
_________________________________________________________________
None



Epoch 1/20
473/473 [==============================] - 73s 140ms/step - loss: 1.4625 - accuracy: 0.2616 - val_loss: 1.3638 - val_accuracy: 0.2447 - lr: 0.0010
Epoch 2/20
473/473 [==============================] - 62s 131ms/step - loss: 1.3958 - accuracy: 0.2665 - val_loss: 1.3371 - val_accuracy: 0.3378 - lr: 0.0010
Epoch 3/20
473/473 [==============================] - 58s 123ms/step - loss: 1.3655 - accuracy: 0.3060 - val_loss: 1.5133 - val_accuracy: 0.1836 - lr: 0.0010
Epoch 4/20
473/473 [==============================] - 58s 122ms/step - loss: 1.3194 - accuracy: 0.3426 - val_loss: 1.5810 - val_accuracy: 0.1943 - lr: 0.0010
Epoch 5/20
473/473 [==============================] - 59s 124ms/step - loss: 1.2924 - accuracy: 0.3683 - val_loss: 1.5175 - val_accuracy: 0.2504 - lr: 0.0010
Epoch 6/20
473/473 [==============================] - 60s 127ms/step - loss: 1.2399 - accuracy: 0.4201 - val_loss: 1.4819 - val_accuracy: 0.3183 - lr: 2.0000e-04
Epoch 7/20
473/473 [==============================] - 61s 129ms/step - loss: 1.2306 - accuracy: 0.4280 - val_loss: 1.2199 - val_accuracy: 0.4354 - lr: 2.0000e-04
Epoch 8/20
473/473 [==============================] - 60s 128ms/step - loss: 1.2215 - accuracy: 0.4339 - val_loss: 1.2098 - val_accuracy: 0.4493 - lr: 2.0000e-04
Epoch 9/20
473/473 [==============================] - 60s 127ms/step - loss: 1.2194 - accuracy: 0.4398 - val_loss: 1.2113 - val_accuracy: 0.4517 - lr: 2.0000e-04
Epoch 10/20
473/473 [==============================] - 58s 123ms/step - loss: 1.2133 - accuracy: 0.4459 - val_loss: 1.3069 - val_accuracy: 0.3908 - lr: 2.0000e-04
Epoch 11/20
473/473 [==============================] - 61s 128ms/step - loss: 1.2091 - accuracy: 0.4492 - val_loss: 1.1824 - val_accuracy: 0.4678 - lr: 2.0000e-04
Epoch 12/20
473/473 [==============================] - 59s 125ms/step - loss: 1.2051 - accuracy: 0.4526 - val_loss: 1.1814 - val_accuracy: 0.4671 - lr: 2.0000e-04
Epoch 13/20
473/473 [==============================] - 61s 128ms/step - loss: 1.2002 - accuracy: 0.4555 - val_loss: 1.1783 - val_accuracy: 0.4778 - lr: 2.0000e-04
Epoch 14/20
473/473 [==============================] - 58s 122ms/step - loss: 1.1955 - accuracy: 0.4554 - val_loss: 1.1905 - val_accuracy: 0.4669 - lr: 2.0000e-04
Epoch 15/20
473/473 [==============================] - 58s 122ms/step - loss: 1.1942 - accuracy: 0.4567 - val_loss: 1.3811 - val_accuracy: 0.3641 - lr: 2.0000e-04
Epoch 16/20
473/473 [==============================] - 58s 123ms/step - loss: 1.1909 - accuracy: 0.4618 - val_loss: 1.1910 - val_accuracy: 0.4736 - lr: 2.0000e-04
Epoch 17/20
473/473 [==============================] - 58s 124ms/step - loss: 1.1796 - accuracy: 0.4757 - val_loss: 1.1884 - val_accuracy: 0.4643 - lr: 4.0000e-05
Epoch 18/20
473/473 [==============================] - 58s 122ms/step - loss: 1.1780 - accuracy: 0.4677 - val_loss: 1.2073 - val_accuracy: 0.4541 - lr: 4.0000e-05



1/1 [==============================] - 0s 111ms/step - loss: 1.1514 - accuracy: 0.5312
1/1 [==============================] - 2s 2s/step
Actual    : [0 1 3 0 1 3 2 1 0 2 0 3 2 2 3 0 3 2 0 2 1 3 2 2 0 0 3 0 1 2 0 2]
Prediction: [0 1 3 2 1 3 2 1 0 2 2 2 2 2 0 0 3 1 0 0 2 2 1 1 0 2 0 0 2 0 0 0]



              precision    recall  f1-score   support

       happy       0.58      0.70      0.64        10
     neutral       0.50      0.60      0.55         5
         sad       0.36      0.40      0.38        10
    surprise       1.00      0.43      0.60         7

    accuracy                           0.53        32
   macro avg       0.61      0.53      0.54        32
weighted avg       0.59      0.53      0.53        32





In [58]:
# Let's try again tuning some of the last layers in the transfer model
batch_size = 32
model, results, history = model_building_and_evaluating_process(model_builder_resnet, f'Model 4: Tuned Resnet V2 ({batch_size})',
                                                                batch_size=batch_size, epochs=20, tune=5, include_grayscale=False)
model_resnet.update(model)
metric_resnet = pd.concat([metric_resnet, results])
history_resnet.update(history)
---------------------------------------------
Model 4: Tuned Resnet V2 (32) - RGB
---------------------------------------------
Initializing TF Session and Seed

Data with color_mode=rgb:
Found 15109 images belonging to 4 classes.
Found 127 images belonging to 4 classes.
Found 4977 images belonging to 4 classes.



Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 resnet101 (Functional)      (None, 2, 2, 2048)        42658176  
                                                                 
 flatten (Flatten)           (None, 8192)              0         
                                                                 
 dense (Dense)               (None, 256)               2097408   
                                                                 
 dense_1 (Dense)             (None, 128)               32896     
                                                                 
 dropout (Dropout)           (None, 128)               0         
                                                                 
 dense_2 (Dense)             (None, 64)                8256      
                                                                 
 batch_normalization (BatchN  (None, 64)               256       
 ormalization)                                                   
                                                                 
 dense_3 (Dense)             (None, 4)                 260       
                                                                 
=================================================================
Total params: 44,797,252
Trainable params: 3,193,668
Non-trainable params: 41,603,584
_________________________________________________________________
None



Epoch 1/20
473/473 [==============================] - 72s 136ms/step - loss: 1.4439 - accuracy: 0.2798 - val_loss: 1.3643 - val_accuracy: 0.2339 - lr: 0.0010
Epoch 2/20
473/473 [==============================] - 62s 131ms/step - loss: 1.3400 - accuracy: 0.3270 - val_loss: 1.4209 - val_accuracy: 0.2522 - lr: 0.0010
Epoch 3/20
473/473 [==============================] - 62s 131ms/step - loss: 1.3090 - accuracy: 0.3579 - val_loss: 1.3305 - val_accuracy: 0.3390 - lr: 0.0010
Epoch 4/20
473/473 [==============================] - 60s 126ms/step - loss: 1.2867 - accuracy: 0.3843 - val_loss: 1.4322 - val_accuracy: 0.2518 - lr: 0.0010
Epoch 5/20
473/473 [==============================] - 59s 125ms/step - loss: 1.2718 - accuracy: 0.3998 - val_loss: 1.3774 - val_accuracy: 0.3138 - lr: 0.0010
Epoch 6/20
473/473 [==============================] - 62s 132ms/step - loss: 1.2544 - accuracy: 0.4141 - val_loss: 1.3111 - val_accuracy: 0.3659 - lr: 0.0010
Epoch 7/20
473/473 [==============================] - 62s 132ms/step - loss: 1.2385 - accuracy: 0.4211 - val_loss: 1.2145 - val_accuracy: 0.4400 - lr: 0.0010
Epoch 8/20
473/473 [==============================] - 61s 128ms/step - loss: 1.2297 - accuracy: 0.4289 - val_loss: 1.3236 - val_accuracy: 0.3500 - lr: 0.0010
Epoch 9/20
473/473 [==============================] - 60s 127ms/step - loss: 1.2218 - accuracy: 0.4342 - val_loss: 1.5070 - val_accuracy: 0.3516 - lr: 0.0010
Epoch 10/20
473/473 [==============================] - 62s 132ms/step - loss: 1.2037 - accuracy: 0.4498 - val_loss: 1.1793 - val_accuracy: 0.4601 - lr: 0.0010
Epoch 11/20
473/473 [==============================] - 62s 131ms/step - loss: 1.1928 - accuracy: 0.4614 - val_loss: 1.2029 - val_accuracy: 0.4641 - lr: 0.0010
Epoch 12/20
473/473 [==============================] - 60s 128ms/step - loss: 1.1891 - accuracy: 0.4591 - val_loss: 1.4216 - val_accuracy: 0.4089 - lr: 0.0010
Epoch 13/20
473/473 [==============================] - 60s 127ms/step - loss: 1.1772 - accuracy: 0.4688 - val_loss: 1.4118 - val_accuracy: 0.3667 - lr: 0.0010
Epoch 14/20
473/473 [==============================] - 62s 131ms/step - loss: 1.1333 - accuracy: 0.5008 - val_loss: 1.1229 - val_accuracy: 0.4967 - lr: 2.0000e-04
Epoch 15/20
473/473 [==============================] - 62s 131ms/step - loss: 1.1315 - accuracy: 0.5072 - val_loss: 1.1263 - val_accuracy: 0.5011 - lr: 2.0000e-04
Epoch 16/20
473/473 [==============================] - 60s 127ms/step - loss: 1.1244 - accuracy: 0.5091 - val_loss: 1.1475 - val_accuracy: 0.4895 - lr: 2.0000e-04
Epoch 17/20
473/473 [==============================] - 62s 130ms/step - loss: 1.1192 - accuracy: 0.5137 - val_loss: 1.1321 - val_accuracy: 0.5067 - lr: 2.0000e-04
Epoch 18/20
473/473 [==============================] - 60s 127ms/step - loss: 1.1115 - accuracy: 0.5130 - val_loss: 1.1718 - val_accuracy: 0.4657 - lr: 4.0000e-05
Epoch 19/20
473/473 [==============================] - 59s 125ms/step - loss: 1.1112 - accuracy: 0.5166 - val_loss: 1.1363 - val_accuracy: 0.4999 - lr: 4.0000e-05



1/1 [==============================] - 0s 113ms/step - loss: 1.1130 - accuracy: 0.3750
1/1 [==============================] - 2s 2s/step
Actual    : [0 1 3 0 1 3 2 1 0 2 0 3 2 2 3 0 3 2 0 2 1 3 2 2 0 0 3 0 1 2 0 2]
Prediction: [0 0 3 1 1 0 1 1 0 3 2 2 2 0 1 0 3 1 0 0 0 2 1 1 0 2 0 1 1 0 0 0]



              precision    recall  f1-score   support

       happy       0.43      0.60      0.50        10
     neutral       0.30      0.60      0.40         5
         sad       0.20      0.10      0.13        10
    surprise       0.67      0.29      0.40         7

    accuracy                           0.38        32
   macro avg       0.40      0.40      0.36        32
weighted avg       0.39      0.38      0.35        32





In [59]:
model_name = 'Model 4: Resnet V2'
save_object(history_resnet, f'{MODEL_DIR}/{model_name} - history.pkl')
save_object(metric_resnet, f'{MODEL_DIR}/{model_name} - metrics.pkl')
save_object(model_resnet, f'{MODEL_DIR}/{model_name} - models.pkl')
In [60]:
loss_and_accuracy_comparisson(history_resnet, data_type='Training Set')
loss_and_accuracy_comparisson(history_resnet, data_type='Validation Set')
In [61]:
metric_resnet.style.highlight_max(color = "lightgreen", axis = 0)
Out[61]:
  Accuracy Train Accuracy Val Accuracy Test
Model 4: Resnet V2 (32) - RGB 0.455490 0.477798 0.531250
Model 4: Tuned Resnet V2 (32) - RGB 0.513667 0.506731 0.375000

Note: You can even go back and build your own architecture on top of the ResNet Transfer layer and see if you can improve the performance.

⏩ Observations and Insights:

  • The Resnet V2 is a convolutional neural network model that's used for image classification. This neural network was trained on more than a million images from the ImageNet database.
  • For the current approach in model no. 4, it was taken from Resnet V2 the convolutional layers block (546 layers), and joined to the classification layers composed mainly of Dense layers, the idea was to verify if this approach improves the results obtained with the previous models.
  • On the first try, it was turned the trainable property of all the convolutional layers (extracted from Resnet V2), to False, so it was ensured that the weight learned in Resnet V2 will be applied to the new problem. This provided 47.8% of accuracy, not good at all.
  • In the second try, the trainable property = False was not applied to all convolutional layers and when the model was compiled it allows the backpropagation to update the last 5 pre-trained layers. The 5 last convolutional layers were trained with the data to help the model learn specific features from the data in this specific image classification problem. This second try achieved 50.7% of accuracy. This tuning improves the performance of the model. Probably a bigger tune number will provide better results. However the results in the testing set are very lower.
  • The learning curve reflects a lot of peaks in the validation performance, maybe because of overfitting. It is necessary to explore if adding Dropout layers just after the convolutional layers could help to deal with this problem.
  • Happy and Surprise are the best-evaluated classes. Neutral and Sad are the classes with the lowest performance.
  • The best model "Model 4: Tuned Resnet V2 (32) - RGB" achieved 50.7% accuracy.

⏩ Recommendations:

  • Increase the number of convolutional layers to retrain with the data, by increasing the tune parameter in the created method "model_builder_resnet".
  • Technique to deal with the overfitting needs to be applied to the model "Model 4: Tuned Resnet V2 (32) - RGB" and see if this improves the result in the validation set.
  • Explore other image augmentation techniques to provide more data to the model.
  • A review of the training dataset is necessary to confirm data are good enough.

Model 5: EfficientNet¶


  • Import EfficientNet upto the layer of your choice and add Fully Connected layers on top of it.
In [62]:
ap.EfficientNetV2B2(include_top=False, weights="imagenet", input_shape=(48, 48, 3)).summary()
Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/efficientnet_v2/efficientnetv2-b2_notop.h5
35839040/35839040 [==============================] - 1s 0us/step
Model: "efficientnetv2-b2"
__________________________________________________________________________________________________
 Layer (type)                   Output Shape         Param #     Connected to                     
==================================================================================================
 input_2 (InputLayer)           [(None, 48, 48, 3)]  0           []                               
                                                                                                  
 rescaling (Rescaling)          (None, 48, 48, 3)    0           ['input_2[0][0]']                
                                                                                                  
 normalization (Normalization)  (None, 48, 48, 3)    0           ['rescaling[0][0]']              
                                                                                                  
 stem_conv (Conv2D)             (None, 24, 24, 32)   864         ['normalization[0][0]']          
                                                                                                  
 stem_bn (BatchNormalization)   (None, 24, 24, 32)   128         ['stem_conv[0][0]']              
                                                                                                  
 stem_activation (Activation)   (None, 24, 24, 32)   0           ['stem_bn[0][0]']                
                                                                                                  
 block1a_project_conv (Conv2D)  (None, 24, 24, 16)   4608        ['stem_activation[0][0]']        
                                                                                                  
 block1a_project_bn (BatchNorma  (None, 24, 24, 16)  64          ['block1a_project_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 block1a_project_activation (Ac  (None, 24, 24, 16)  0           ['block1a_project_bn[0][0]']     
 tivation)                                                                                        
                                                                                                  
 block1b_project_conv (Conv2D)  (None, 24, 24, 16)   2304        ['block1a_project_activation[0][0
                                                                 ]']                              
                                                                                                  
 block1b_project_bn (BatchNorma  (None, 24, 24, 16)  64          ['block1b_project_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 block1b_project_activation (Ac  (None, 24, 24, 16)  0           ['block1b_project_bn[0][0]']     
 tivation)                                                                                        
                                                                                                  
 block1b_drop (Dropout)         (None, 24, 24, 16)   0           ['block1b_project_activation[0][0
                                                                 ]']                              
                                                                                                  
 block1b_add (Add)              (None, 24, 24, 16)   0           ['block1b_drop[0][0]',           
                                                                  'block1a_project_activation[0][0
                                                                 ]']                              
                                                                                                  
 block2a_expand_conv (Conv2D)   (None, 12, 12, 64)   9216        ['block1b_add[0][0]']            
                                                                                                  
 block2a_expand_bn (BatchNormal  (None, 12, 12, 64)  256         ['block2a_expand_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 block2a_expand_activation (Act  (None, 12, 12, 64)  0           ['block2a_expand_bn[0][0]']      
 ivation)                                                                                         
                                                                                                  
 block2a_project_conv (Conv2D)  (None, 12, 12, 32)   2048        ['block2a_expand_activation[0][0]
                                                                 ']                               
                                                                                                  
 block2a_project_bn (BatchNorma  (None, 12, 12, 32)  128         ['block2a_project_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 block2b_expand_conv (Conv2D)   (None, 12, 12, 128)  36864       ['block2a_project_bn[0][0]']     
                                                                                                  
 block2b_expand_bn (BatchNormal  (None, 12, 12, 128)  512        ['block2b_expand_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 block2b_expand_activation (Act  (None, 12, 12, 128)  0          ['block2b_expand_bn[0][0]']      
 ivation)                                                                                         
                                                                                                  
 block2b_project_conv (Conv2D)  (None, 12, 12, 32)   4096        ['block2b_expand_activation[0][0]
                                                                 ']                               
                                                                                                  
 block2b_project_bn (BatchNorma  (None, 12, 12, 32)  128         ['block2b_project_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 block2b_drop (Dropout)         (None, 12, 12, 32)   0           ['block2b_project_bn[0][0]']     
                                                                                                  
 block2b_add (Add)              (None, 12, 12, 32)   0           ['block2b_drop[0][0]',           
                                                                  'block2a_project_bn[0][0]']     
                                                                                                  
 block2c_expand_conv (Conv2D)   (None, 12, 12, 128)  36864       ['block2b_add[0][0]']            
                                                                                                  
 block2c_expand_bn (BatchNormal  (None, 12, 12, 128)  512        ['block2c_expand_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 block2c_expand_activation (Act  (None, 12, 12, 128)  0          ['block2c_expand_bn[0][0]']      
 ivation)                                                                                         
                                                                                                  
 block2c_project_conv (Conv2D)  (None, 12, 12, 32)   4096        ['block2c_expand_activation[0][0]
                                                                 ']                               
                                                                                                  
 block2c_project_bn (BatchNorma  (None, 12, 12, 32)  128         ['block2c_project_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 block2c_drop (Dropout)         (None, 12, 12, 32)   0           ['block2c_project_bn[0][0]']     
                                                                                                  
 block2c_add (Add)              (None, 12, 12, 32)   0           ['block2c_drop[0][0]',           
                                                                  'block2b_add[0][0]']            
                                                                                                  
 block3a_expand_conv (Conv2D)   (None, 6, 6, 128)    36864       ['block2c_add[0][0]']            
                                                                                                  
 block3a_expand_bn (BatchNormal  (None, 6, 6, 128)   512         ['block3a_expand_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 block3a_expand_activation (Act  (None, 6, 6, 128)   0           ['block3a_expand_bn[0][0]']      
 ivation)                                                                                         
                                                                                                  
 block3a_project_conv (Conv2D)  (None, 6, 6, 56)     7168        ['block3a_expand_activation[0][0]
                                                                 ']                               
                                                                                                  
 block3a_project_bn (BatchNorma  (None, 6, 6, 56)    224         ['block3a_project_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 block3b_expand_conv (Conv2D)   (None, 6, 6, 224)    112896      ['block3a_project_bn[0][0]']     
                                                                                                  
 block3b_expand_bn (BatchNormal  (None, 6, 6, 224)   896         ['block3b_expand_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 block3b_expand_activation (Act  (None, 6, 6, 224)   0           ['block3b_expand_bn[0][0]']      
 ivation)                                                                                         
                                                                                                  
 block3b_project_conv (Conv2D)  (None, 6, 6, 56)     12544       ['block3b_expand_activation[0][0]
                                                                 ']                               
                                                                                                  
 block3b_project_bn (BatchNorma  (None, 6, 6, 56)    224         ['block3b_project_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 block3b_drop (Dropout)         (None, 6, 6, 56)     0           ['block3b_project_bn[0][0]']     
                                                                                                  
 block3b_add (Add)              (None, 6, 6, 56)     0           ['block3b_drop[0][0]',           
                                                                  'block3a_project_bn[0][0]']     
                                                                                                  
 block3c_expand_conv (Conv2D)   (None, 6, 6, 224)    112896      ['block3b_add[0][0]']            
                                                                                                  
 block3c_expand_bn (BatchNormal  (None, 6, 6, 224)   896         ['block3c_expand_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 block3c_expand_activation (Act  (None, 6, 6, 224)   0           ['block3c_expand_bn[0][0]']      
 ivation)                                                                                         
                                                                                                  
 block3c_project_conv (Conv2D)  (None, 6, 6, 56)     12544       ['block3c_expand_activation[0][0]
                                                                 ']                               
                                                                                                  
 block3c_project_bn (BatchNorma  (None, 6, 6, 56)    224         ['block3c_project_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 block3c_drop (Dropout)         (None, 6, 6, 56)     0           ['block3c_project_bn[0][0]']     
                                                                                                  
 block3c_add (Add)              (None, 6, 6, 56)     0           ['block3c_drop[0][0]',           
                                                                  'block3b_add[0][0]']            
                                                                                                  
 block4a_expand_conv (Conv2D)   (None, 6, 6, 224)    12544       ['block3c_add[0][0]']            
                                                                                                  
 block4a_expand_bn (BatchNormal  (None, 6, 6, 224)   896         ['block4a_expand_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 block4a_expand_activation (Act  (None, 6, 6, 224)   0           ['block4a_expand_bn[0][0]']      
 ivation)                                                                                         
                                                                                                  
 block4a_dwconv2 (DepthwiseConv  (None, 3, 3, 224)   2016        ['block4a_expand_activation[0][0]
 2D)                                                             ']                               
                                                                                                  
 block4a_bn (BatchNormalization  (None, 3, 3, 224)   896         ['block4a_dwconv2[0][0]']        
 )                                                                                                
                                                                                                  
 block4a_activation (Activation  (None, 3, 3, 224)   0           ['block4a_bn[0][0]']             
 )                                                                                                
                                                                                                  
 block4a_se_squeeze (GlobalAver  (None, 224)         0           ['block4a_activation[0][0]']     
 agePooling2D)                                                                                    
                                                                                                  
 block4a_se_reshape (Reshape)   (None, 1, 1, 224)    0           ['block4a_se_squeeze[0][0]']     
                                                                                                  
 block4a_se_reduce (Conv2D)     (None, 1, 1, 14)     3150        ['block4a_se_reshape[0][0]']     
                                                                                                  
 block4a_se_expand (Conv2D)     (None, 1, 1, 224)    3360        ['block4a_se_reduce[0][0]']      
                                                                                                  
 block4a_se_excite (Multiply)   (None, 3, 3, 224)    0           ['block4a_activation[0][0]',     
                                                                  'block4a_se_expand[0][0]']      
                                                                                                  
 block4a_project_conv (Conv2D)  (None, 3, 3, 104)    23296       ['block4a_se_excite[0][0]']      
                                                                                                  
 block4a_project_bn (BatchNorma  (None, 3, 3, 104)   416         ['block4a_project_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 block4b_expand_conv (Conv2D)   (None, 3, 3, 416)    43264       ['block4a_project_bn[0][0]']     
                                                                                                  
 block4b_expand_bn (BatchNormal  (None, 3, 3, 416)   1664        ['block4b_expand_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 block4b_expand_activation (Act  (None, 3, 3, 416)   0           ['block4b_expand_bn[0][0]']      
 ivation)                                                                                         
                                                                                                  
 block4b_dwconv2 (DepthwiseConv  (None, 3, 3, 416)   3744        ['block4b_expand_activation[0][0]
 2D)                                                             ']                               
                                                                                                  
 block4b_bn (BatchNormalization  (None, 3, 3, 416)   1664        ['block4b_dwconv2[0][0]']        
 )                                                                                                
                                                                                                  
 block4b_activation (Activation  (None, 3, 3, 416)   0           ['block4b_bn[0][0]']             
 )                                                                                                
                                                                                                  
 block4b_se_squeeze (GlobalAver  (None, 416)         0           ['block4b_activation[0][0]']     
 agePooling2D)                                                                                    
                                                                                                  
 block4b_se_reshape (Reshape)   (None, 1, 1, 416)    0           ['block4b_se_squeeze[0][0]']     
                                                                                                  
 block4b_se_reduce (Conv2D)     (None, 1, 1, 26)     10842       ['block4b_se_reshape[0][0]']     
                                                                                                  
 block4b_se_expand (Conv2D)     (None, 1, 1, 416)    11232       ['block4b_se_reduce[0][0]']      
                                                                                                  
 block4b_se_excite (Multiply)   (None, 3, 3, 416)    0           ['block4b_activation[0][0]',     
                                                                  'block4b_se_expand[0][0]']      
                                                                                                  
 block4b_project_conv (Conv2D)  (None, 3, 3, 104)    43264       ['block4b_se_excite[0][0]']      
                                                                                                  
 block4b_project_bn (BatchNorma  (None, 3, 3, 104)   416         ['block4b_project_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 block4b_drop (Dropout)         (None, 3, 3, 104)    0           ['block4b_project_bn[0][0]']     
                                                                                                  
 block4b_add (Add)              (None, 3, 3, 104)    0           ['block4b_drop[0][0]',           
                                                                  'block4a_project_bn[0][0]']     
                                                                                                  
 block4c_expand_conv (Conv2D)   (None, 3, 3, 416)    43264       ['block4b_add[0][0]']            
                                                                                                  
 block4c_expand_bn (BatchNormal  (None, 3, 3, 416)   1664        ['block4c_expand_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 block4c_expand_activation (Act  (None, 3, 3, 416)   0           ['block4c_expand_bn[0][0]']      
 ivation)                                                                                         
                                                                                                  
 block4c_dwconv2 (DepthwiseConv  (None, 3, 3, 416)   3744        ['block4c_expand_activation[0][0]
 2D)                                                             ']                               
                                                                                                  
 block4c_bn (BatchNormalization  (None, 3, 3, 416)   1664        ['block4c_dwconv2[0][0]']        
 )                                                                                                
                                                                                                  
 block4c_activation (Activation  (None, 3, 3, 416)   0           ['block4c_bn[0][0]']             
 )                                                                                                
                                                                                                  
 block4c_se_squeeze (GlobalAver  (None, 416)         0           ['block4c_activation[0][0]']     
 agePooling2D)                                                                                    
                                                                                                  
 block4c_se_reshape (Reshape)   (None, 1, 1, 416)    0           ['block4c_se_squeeze[0][0]']     
                                                                                                  
 block4c_se_reduce (Conv2D)     (None, 1, 1, 26)     10842       ['block4c_se_reshape[0][0]']     
                                                                                                  
 block4c_se_expand (Conv2D)     (None, 1, 1, 416)    11232       ['block4c_se_reduce[0][0]']      
                                                                                                  
 block4c_se_excite (Multiply)   (None, 3, 3, 416)    0           ['block4c_activation[0][0]',     
                                                                  'block4c_se_expand[0][0]']      
                                                                                                  
 block4c_project_conv (Conv2D)  (None, 3, 3, 104)    43264       ['block4c_se_excite[0][0]']      
                                                                                                  
 block4c_project_bn (BatchNorma  (None, 3, 3, 104)   416         ['block4c_project_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 block4c_drop (Dropout)         (None, 3, 3, 104)    0           ['block4c_project_bn[0][0]']     
                                                                                                  
 block4c_add (Add)              (None, 3, 3, 104)    0           ['block4c_drop[0][0]',           
                                                                  'block4b_add[0][0]']            
                                                                                                  
 block4d_expand_conv (Conv2D)   (None, 3, 3, 416)    43264       ['block4c_add[0][0]']            
                                                                                                  
 block4d_expand_bn (BatchNormal  (None, 3, 3, 416)   1664        ['block4d_expand_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 block4d_expand_activation (Act  (None, 3, 3, 416)   0           ['block4d_expand_bn[0][0]']      
 ivation)                                                                                         
                                                                                                  
 block4d_dwconv2 (DepthwiseConv  (None, 3, 3, 416)   3744        ['block4d_expand_activation[0][0]
 2D)                                                             ']                               
                                                                                                  
 block4d_bn (BatchNormalization  (None, 3, 3, 416)   1664        ['block4d_dwconv2[0][0]']        
 )                                                                                                
                                                                                                  
 block4d_activation (Activation  (None, 3, 3, 416)   0           ['block4d_bn[0][0]']             
 )                                                                                                
                                                                                                  
 block4d_se_squeeze (GlobalAver  (None, 416)         0           ['block4d_activation[0][0]']     
 agePooling2D)                                                                                    
                                                                                                  
 block4d_se_reshape (Reshape)   (None, 1, 1, 416)    0           ['block4d_se_squeeze[0][0]']     
                                                                                                  
 block4d_se_reduce (Conv2D)     (None, 1, 1, 26)     10842       ['block4d_se_reshape[0][0]']     
                                                                                                  
 block4d_se_expand (Conv2D)     (None, 1, 1, 416)    11232       ['block4d_se_reduce[0][0]']      
                                                                                                  
 block4d_se_excite (Multiply)   (None, 3, 3, 416)    0           ['block4d_activation[0][0]',     
                                                                  'block4d_se_expand[0][0]']      
                                                                                                  
 block4d_project_conv (Conv2D)  (None, 3, 3, 104)    43264       ['block4d_se_excite[0][0]']      
                                                                                                  
 block4d_project_bn (BatchNorma  (None, 3, 3, 104)   416         ['block4d_project_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 block4d_drop (Dropout)         (None, 3, 3, 104)    0           ['block4d_project_bn[0][0]']     
                                                                                                  
 block4d_add (Add)              (None, 3, 3, 104)    0           ['block4d_drop[0][0]',           
                                                                  'block4c_add[0][0]']            
                                                                                                  
 block5a_expand_conv (Conv2D)   (None, 3, 3, 624)    64896       ['block4d_add[0][0]']            
                                                                                                  
 block5a_expand_bn (BatchNormal  (None, 3, 3, 624)   2496        ['block5a_expand_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 block5a_expand_activation (Act  (None, 3, 3, 624)   0           ['block5a_expand_bn[0][0]']      
 ivation)                                                                                         
                                                                                                  
 block5a_dwconv2 (DepthwiseConv  (None, 3, 3, 624)   5616        ['block5a_expand_activation[0][0]
 2D)                                                             ']                               
                                                                                                  
 block5a_bn (BatchNormalization  (None, 3, 3, 624)   2496        ['block5a_dwconv2[0][0]']        
 )                                                                                                
                                                                                                  
 block5a_activation (Activation  (None, 3, 3, 624)   0           ['block5a_bn[0][0]']             
 )                                                                                                
                                                                                                  
 block5a_se_squeeze (GlobalAver  (None, 624)         0           ['block5a_activation[0][0]']     
 agePooling2D)                                                                                    
                                                                                                  
 block5a_se_reshape (Reshape)   (None, 1, 1, 624)    0           ['block5a_se_squeeze[0][0]']     
                                                                                                  
 block5a_se_reduce (Conv2D)     (None, 1, 1, 26)     16250       ['block5a_se_reshape[0][0]']     
                                                                                                  
 block5a_se_expand (Conv2D)     (None, 1, 1, 624)    16848       ['block5a_se_reduce[0][0]']      
                                                                                                  
 block5a_se_excite (Multiply)   (None, 3, 3, 624)    0           ['block5a_activation[0][0]',     
                                                                  'block5a_se_expand[0][0]']      
                                                                                                  
 block5a_project_conv (Conv2D)  (None, 3, 3, 120)    74880       ['block5a_se_excite[0][0]']      
                                                                                                  
 block5a_project_bn (BatchNorma  (None, 3, 3, 120)   480         ['block5a_project_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 block5b_expand_conv (Conv2D)   (None, 3, 3, 720)    86400       ['block5a_project_bn[0][0]']     
                                                                                                  
 block5b_expand_bn (BatchNormal  (None, 3, 3, 720)   2880        ['block5b_expand_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 block5b_expand_activation (Act  (None, 3, 3, 720)   0           ['block5b_expand_bn[0][0]']      
 ivation)                                                                                         
                                                                                                  
 block5b_dwconv2 (DepthwiseConv  (None, 3, 3, 720)   6480        ['block5b_expand_activation[0][0]
 2D)                                                             ']                               
                                                                                                  
 block5b_bn (BatchNormalization  (None, 3, 3, 720)   2880        ['block5b_dwconv2[0][0]']        
 )                                                                                                
                                                                                                  
 block5b_activation (Activation  (None, 3, 3, 720)   0           ['block5b_bn[0][0]']             
 )                                                                                                
                                                                                                  
 block5b_se_squeeze (GlobalAver  (None, 720)         0           ['block5b_activation[0][0]']     
 agePooling2D)                                                                                    
                                                                                                  
 block5b_se_reshape (Reshape)   (None, 1, 1, 720)    0           ['block5b_se_squeeze[0][0]']     
                                                                                                  
 block5b_se_reduce (Conv2D)     (None, 1, 1, 30)     21630       ['block5b_se_reshape[0][0]']     
                                                                                                  
 block5b_se_expand (Conv2D)     (None, 1, 1, 720)    22320       ['block5b_se_reduce[0][0]']      
                                                                                                  
 block5b_se_excite (Multiply)   (None, 3, 3, 720)    0           ['block5b_activation[0][0]',     
                                                                  'block5b_se_expand[0][0]']      
                                                                                                  
 block5b_project_conv (Conv2D)  (None, 3, 3, 120)    86400       ['block5b_se_excite[0][0]']      
                                                                                                  
 block5b_project_bn (BatchNorma  (None, 3, 3, 120)   480         ['block5b_project_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 block5b_drop (Dropout)         (None, 3, 3, 120)    0           ['block5b_project_bn[0][0]']     
                                                                                                  
 block5b_add (Add)              (None, 3, 3, 120)    0           ['block5b_drop[0][0]',           
                                                                  'block5a_project_bn[0][0]']     
                                                                                                  
 block5c_expand_conv (Conv2D)   (None, 3, 3, 720)    86400       ['block5b_add[0][0]']            
                                                                                                  
 block5c_expand_bn (BatchNormal  (None, 3, 3, 720)   2880        ['block5c_expand_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 block5c_expand_activation (Act  (None, 3, 3, 720)   0           ['block5c_expand_bn[0][0]']      
 ivation)                                                                                         
                                                                                                  
 block5c_dwconv2 (DepthwiseConv  (None, 3, 3, 720)   6480        ['block5c_expand_activation[0][0]
 2D)                                                             ']                               
                                                                                                  
 block5c_bn (BatchNormalization  (None, 3, 3, 720)   2880        ['block5c_dwconv2[0][0]']        
 )                                                                                                
                                                                                                  
 block5c_activation (Activation  (None, 3, 3, 720)   0           ['block5c_bn[0][0]']             
 )                                                                                                
                                                                                                  
 block5c_se_squeeze (GlobalAver  (None, 720)         0           ['block5c_activation[0][0]']     
 agePooling2D)                                                                                    
                                                                                                  
 block5c_se_reshape (Reshape)   (None, 1, 1, 720)    0           ['block5c_se_squeeze[0][0]']     
                                                                                                  
 block5c_se_reduce (Conv2D)     (None, 1, 1, 30)     21630       ['block5c_se_reshape[0][0]']     
                                                                                                  
 block5c_se_expand (Conv2D)     (None, 1, 1, 720)    22320       ['block5c_se_reduce[0][0]']      
                                                                                                  
 block5c_se_excite (Multiply)   (None, 3, 3, 720)    0           ['block5c_activation[0][0]',     
                                                                  'block5c_se_expand[0][0]']      
                                                                                                  
 block5c_project_conv (Conv2D)  (None, 3, 3, 120)    86400       ['block5c_se_excite[0][0]']      
                                                                                                  
 block5c_project_bn (BatchNorma  (None, 3, 3, 120)   480         ['block5c_project_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 block5c_drop (Dropout)         (None, 3, 3, 120)    0           ['block5c_project_bn[0][0]']     
                                                                                                  
 block5c_add (Add)              (None, 3, 3, 120)    0           ['block5c_drop[0][0]',           
                                                                  'block5b_add[0][0]']            
                                                                                                  
 block5d_expand_conv (Conv2D)   (None, 3, 3, 720)    86400       ['block5c_add[0][0]']            
                                                                                                  
 block5d_expand_bn (BatchNormal  (None, 3, 3, 720)   2880        ['block5d_expand_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 block5d_expand_activation (Act  (None, 3, 3, 720)   0           ['block5d_expand_bn[0][0]']      
 ivation)                                                                                         
                                                                                                  
 block5d_dwconv2 (DepthwiseConv  (None, 3, 3, 720)   6480        ['block5d_expand_activation[0][0]
 2D)                                                             ']                               
                                                                                                  
 block5d_bn (BatchNormalization  (None, 3, 3, 720)   2880        ['block5d_dwconv2[0][0]']        
 )                                                                                                
                                                                                                  
 block5d_activation (Activation  (None, 3, 3, 720)   0           ['block5d_bn[0][0]']             
 )                                                                                                
                                                                                                  
 block5d_se_squeeze (GlobalAver  (None, 720)         0           ['block5d_activation[0][0]']     
 agePooling2D)                                                                                    
                                                                                                  
 block5d_se_reshape (Reshape)   (None, 1, 1, 720)    0           ['block5d_se_squeeze[0][0]']     
                                                                                                  
 block5d_se_reduce (Conv2D)     (None, 1, 1, 30)     21630       ['block5d_se_reshape[0][0]']     
                                                                                                  
 block5d_se_expand (Conv2D)     (None, 1, 1, 720)    22320       ['block5d_se_reduce[0][0]']      
                                                                                                  
 block5d_se_excite (Multiply)   (None, 3, 3, 720)    0           ['block5d_activation[0][0]',     
                                                                  'block5d_se_expand[0][0]']      
                                                                                                  
 block5d_project_conv (Conv2D)  (None, 3, 3, 120)    86400       ['block5d_se_excite[0][0]']      
                                                                                                  
 block5d_project_bn (BatchNorma  (None, 3, 3, 120)   480         ['block5d_project_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 block5d_drop (Dropout)         (None, 3, 3, 120)    0           ['block5d_project_bn[0][0]']     
                                                                                                  
 block5d_add (Add)              (None, 3, 3, 120)    0           ['block5d_drop[0][0]',           
                                                                  'block5c_add[0][0]']            
                                                                                                  
 block5e_expand_conv (Conv2D)   (None, 3, 3, 720)    86400       ['block5d_add[0][0]']            
                                                                                                  
 block5e_expand_bn (BatchNormal  (None, 3, 3, 720)   2880        ['block5e_expand_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 block5e_expand_activation (Act  (None, 3, 3, 720)   0           ['block5e_expand_bn[0][0]']      
 ivation)                                                                                         
                                                                                                  
 block5e_dwconv2 (DepthwiseConv  (None, 3, 3, 720)   6480        ['block5e_expand_activation[0][0]
 2D)                                                             ']                               
                                                                                                  
 block5e_bn (BatchNormalization  (None, 3, 3, 720)   2880        ['block5e_dwconv2[0][0]']        
 )                                                                                                
                                                                                                  
 block5e_activation (Activation  (None, 3, 3, 720)   0           ['block5e_bn[0][0]']             
 )                                                                                                
                                                                                                  
 block5e_se_squeeze (GlobalAver  (None, 720)         0           ['block5e_activation[0][0]']     
 agePooling2D)                                                                                    
                                                                                                  
 block5e_se_reshape (Reshape)   (None, 1, 1, 720)    0           ['block5e_se_squeeze[0][0]']     
                                                                                                  
 block5e_se_reduce (Conv2D)     (None, 1, 1, 30)     21630       ['block5e_se_reshape[0][0]']     
                                                                                                  
 block5e_se_expand (Conv2D)     (None, 1, 1, 720)    22320       ['block5e_se_reduce[0][0]']      
                                                                                                  
 block5e_se_excite (Multiply)   (None, 3, 3, 720)    0           ['block5e_activation[0][0]',     
                                                                  'block5e_se_expand[0][0]']      
                                                                                                  
 block5e_project_conv (Conv2D)  (None, 3, 3, 120)    86400       ['block5e_se_excite[0][0]']      
                                                                                                  
 block5e_project_bn (BatchNorma  (None, 3, 3, 120)   480         ['block5e_project_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 block5e_drop (Dropout)         (None, 3, 3, 120)    0           ['block5e_project_bn[0][0]']     
                                                                                                  
 block5e_add (Add)              (None, 3, 3, 120)    0           ['block5e_drop[0][0]',           
                                                                  'block5d_add[0][0]']            
                                                                                                  
 block5f_expand_conv (Conv2D)   (None, 3, 3, 720)    86400       ['block5e_add[0][0]']            
                                                                                                  
 block5f_expand_bn (BatchNormal  (None, 3, 3, 720)   2880        ['block5f_expand_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 block5f_expand_activation (Act  (None, 3, 3, 720)   0           ['block5f_expand_bn[0][0]']      
 ivation)                                                                                         
                                                                                                  
 block5f_dwconv2 (DepthwiseConv  (None, 3, 3, 720)   6480        ['block5f_expand_activation[0][0]
 2D)                                                             ']                               
                                                                                                  
 block5f_bn (BatchNormalization  (None, 3, 3, 720)   2880        ['block5f_dwconv2[0][0]']        
 )                                                                                                
                                                                                                  
 block5f_activation (Activation  (None, 3, 3, 720)   0           ['block5f_bn[0][0]']             
 )                                                                                                
                                                                                                  
 block5f_se_squeeze (GlobalAver  (None, 720)         0           ['block5f_activation[0][0]']     
 agePooling2D)                                                                                    
                                                                                                  
 block5f_se_reshape (Reshape)   (None, 1, 1, 720)    0           ['block5f_se_squeeze[0][0]']     
                                                                                                  
 block5f_se_reduce (Conv2D)     (None, 1, 1, 30)     21630       ['block5f_se_reshape[0][0]']     
                                                                                                  
 block5f_se_expand (Conv2D)     (None, 1, 1, 720)    22320       ['block5f_se_reduce[0][0]']      
                                                                                                  
 block5f_se_excite (Multiply)   (None, 3, 3, 720)    0           ['block5f_activation[0][0]',     
                                                                  'block5f_se_expand[0][0]']      
                                                                                                  
 block5f_project_conv (Conv2D)  (None, 3, 3, 120)    86400       ['block5f_se_excite[0][0]']      
                                                                                                  
 block5f_project_bn (BatchNorma  (None, 3, 3, 120)   480         ['block5f_project_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 block5f_drop (Dropout)         (None, 3, 3, 120)    0           ['block5f_project_bn[0][0]']     
                                                                                                  
 block5f_add (Add)              (None, 3, 3, 120)    0           ['block5f_drop[0][0]',           
                                                                  'block5e_add[0][0]']            
                                                                                                  
 block6a_expand_conv (Conv2D)   (None, 3, 3, 720)    86400       ['block5f_add[0][0]']            
                                                                                                  
 block6a_expand_bn (BatchNormal  (None, 3, 3, 720)   2880        ['block6a_expand_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 block6a_expand_activation (Act  (None, 3, 3, 720)   0           ['block6a_expand_bn[0][0]']      
 ivation)                                                                                         
                                                                                                  
 block6a_dwconv2 (DepthwiseConv  (None, 2, 2, 720)   6480        ['block6a_expand_activation[0][0]
 2D)                                                             ']                               
                                                                                                  
 block6a_bn (BatchNormalization  (None, 2, 2, 720)   2880        ['block6a_dwconv2[0][0]']        
 )                                                                                                
                                                                                                  
 block6a_activation (Activation  (None, 2, 2, 720)   0           ['block6a_bn[0][0]']             
 )                                                                                                
                                                                                                  
 block6a_se_squeeze (GlobalAver  (None, 720)         0           ['block6a_activation[0][0]']     
 agePooling2D)                                                                                    
                                                                                                  
 block6a_se_reshape (Reshape)   (None, 1, 1, 720)    0           ['block6a_se_squeeze[0][0]']     
                                                                                                  
 block6a_se_reduce (Conv2D)     (None, 1, 1, 30)     21630       ['block6a_se_reshape[0][0]']     
                                                                                                  
 block6a_se_expand (Conv2D)     (None, 1, 1, 720)    22320       ['block6a_se_reduce[0][0]']      
                                                                                                  
 block6a_se_excite (Multiply)   (None, 2, 2, 720)    0           ['block6a_activation[0][0]',     
                                                                  'block6a_se_expand[0][0]']      
                                                                                                  
 block6a_project_conv (Conv2D)  (None, 2, 2, 208)    149760      ['block6a_se_excite[0][0]']      
                                                                                                  
 block6a_project_bn (BatchNorma  (None, 2, 2, 208)   832         ['block6a_project_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 block6b_expand_conv (Conv2D)   (None, 2, 2, 1248)   259584      ['block6a_project_bn[0][0]']     
                                                                                                  
 block6b_expand_bn (BatchNormal  (None, 2, 2, 1248)  4992        ['block6b_expand_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 block6b_expand_activation (Act  (None, 2, 2, 1248)  0           ['block6b_expand_bn[0][0]']      
 ivation)                                                                                         
                                                                                                  
 block6b_dwconv2 (DepthwiseConv  (None, 2, 2, 1248)  11232       ['block6b_expand_activation[0][0]
 2D)                                                             ']                               
                                                                                                  
 block6b_bn (BatchNormalization  (None, 2, 2, 1248)  4992        ['block6b_dwconv2[0][0]']        
 )                                                                                                
                                                                                                  
 block6b_activation (Activation  (None, 2, 2, 1248)  0           ['block6b_bn[0][0]']             
 )                                                                                                
                                                                                                  
 block6b_se_squeeze (GlobalAver  (None, 1248)        0           ['block6b_activation[0][0]']     
 agePooling2D)                                                                                    
                                                                                                  
 block6b_se_reshape (Reshape)   (None, 1, 1, 1248)   0           ['block6b_se_squeeze[0][0]']     
                                                                                                  
 block6b_se_reduce (Conv2D)     (None, 1, 1, 52)     64948       ['block6b_se_reshape[0][0]']     
                                                                                                  
 block6b_se_expand (Conv2D)     (None, 1, 1, 1248)   66144       ['block6b_se_reduce[0][0]']      
                                                                                                  
 block6b_se_excite (Multiply)   (None, 2, 2, 1248)   0           ['block6b_activation[0][0]',     
                                                                  'block6b_se_expand[0][0]']      
                                                                                                  
 block6b_project_conv (Conv2D)  (None, 2, 2, 208)    259584      ['block6b_se_excite[0][0]']      
                                                                                                  
 block6b_project_bn (BatchNorma  (None, 2, 2, 208)   832         ['block6b_project_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 block6b_drop (Dropout)         (None, 2, 2, 208)    0           ['block6b_project_bn[0][0]']     
                                                                                                  
 block6b_add (Add)              (None, 2, 2, 208)    0           ['block6b_drop[0][0]',           
                                                                  'block6a_project_bn[0][0]']     
                                                                                                  
 block6c_expand_conv (Conv2D)   (None, 2, 2, 1248)   259584      ['block6b_add[0][0]']            
                                                                                                  
 block6c_expand_bn (BatchNormal  (None, 2, 2, 1248)  4992        ['block6c_expand_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 block6c_expand_activation (Act  (None, 2, 2, 1248)  0           ['block6c_expand_bn[0][0]']      
 ivation)                                                                                         
                                                                                                  
 block6c_dwconv2 (DepthwiseConv  (None, 2, 2, 1248)  11232       ['block6c_expand_activation[0][0]
 2D)                                                             ']                               
                                                                                                  
 block6c_bn (BatchNormalization  (None, 2, 2, 1248)  4992        ['block6c_dwconv2[0][0]']        
 )                                                                                                
                                                                                                  
 block6c_activation (Activation  (None, 2, 2, 1248)  0           ['block6c_bn[0][0]']             
 )                                                                                                
                                                                                                  
 block6c_se_squeeze (GlobalAver  (None, 1248)        0           ['block6c_activation[0][0]']     
 agePooling2D)                                                                                    
                                                                                                  
 block6c_se_reshape (Reshape)   (None, 1, 1, 1248)   0           ['block6c_se_squeeze[0][0]']     
                                                                                                  
 block6c_se_reduce (Conv2D)     (None, 1, 1, 52)     64948       ['block6c_se_reshape[0][0]']     
                                                                                                  
 block6c_se_expand (Conv2D)     (None, 1, 1, 1248)   66144       ['block6c_se_reduce[0][0]']      
                                                                                                  
 block6c_se_excite (Multiply)   (None, 2, 2, 1248)   0           ['block6c_activation[0][0]',     
                                                                  'block6c_se_expand[0][0]']      
                                                                                                  
 block6c_project_conv (Conv2D)  (None, 2, 2, 208)    259584      ['block6c_se_excite[0][0]']      
                                                                                                  
 block6c_project_bn (BatchNorma  (None, 2, 2, 208)   832         ['block6c_project_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 block6c_drop (Dropout)         (None, 2, 2, 208)    0           ['block6c_project_bn[0][0]']     
                                                                                                  
 block6c_add (Add)              (None, 2, 2, 208)    0           ['block6c_drop[0][0]',           
                                                                  'block6b_add[0][0]']            
                                                                                                  
 block6d_expand_conv (Conv2D)   (None, 2, 2, 1248)   259584      ['block6c_add[0][0]']            
                                                                                                  
 block6d_expand_bn (BatchNormal  (None, 2, 2, 1248)  4992        ['block6d_expand_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 block6d_expand_activation (Act  (None, 2, 2, 1248)  0           ['block6d_expand_bn[0][0]']      
 ivation)                                                                                         
                                                                                                  
 block6d_dwconv2 (DepthwiseConv  (None, 2, 2, 1248)  11232       ['block6d_expand_activation[0][0]
 2D)                                                             ']                               
                                                                                                  
 block6d_bn (BatchNormalization  (None, 2, 2, 1248)  4992        ['block6d_dwconv2[0][0]']        
 )                                                                                                
                                                                                                  
 block6d_activation (Activation  (None, 2, 2, 1248)  0           ['block6d_bn[0][0]']             
 )                                                                                                
                                                                                                  
 block6d_se_squeeze (GlobalAver  (None, 1248)        0           ['block6d_activation[0][0]']     
 agePooling2D)                                                                                    
                                                                                                  
 block6d_se_reshape (Reshape)   (None, 1, 1, 1248)   0           ['block6d_se_squeeze[0][0]']     
                                                                                                  
 block6d_se_reduce (Conv2D)     (None, 1, 1, 52)     64948       ['block6d_se_reshape[0][0]']     
                                                                                                  
 block6d_se_expand (Conv2D)     (None, 1, 1, 1248)   66144       ['block6d_se_reduce[0][0]']      
                                                                                                  
 block6d_se_excite (Multiply)   (None, 2, 2, 1248)   0           ['block6d_activation[0][0]',     
                                                                  'block6d_se_expand[0][0]']      
                                                                                                  
 block6d_project_conv (Conv2D)  (None, 2, 2, 208)    259584      ['block6d_se_excite[0][0]']      
                                                                                                  
 block6d_project_bn (BatchNorma  (None, 2, 2, 208)   832         ['block6d_project_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 block6d_drop (Dropout)         (None, 2, 2, 208)    0           ['block6d_project_bn[0][0]']     
                                                                                                  
 block6d_add (Add)              (None, 2, 2, 208)    0           ['block6d_drop[0][0]',           
                                                                  'block6c_add[0][0]']            
                                                                                                  
 block6e_expand_conv (Conv2D)   (None, 2, 2, 1248)   259584      ['block6d_add[0][0]']            
                                                                                                  
 block6e_expand_bn (BatchNormal  (None, 2, 2, 1248)  4992        ['block6e_expand_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 block6e_expand_activation (Act  (None, 2, 2, 1248)  0           ['block6e_expand_bn[0][0]']      
 ivation)                                                                                         
                                                                                                  
 block6e_dwconv2 (DepthwiseConv  (None, 2, 2, 1248)  11232       ['block6e_expand_activation[0][0]
 2D)                                                             ']                               
                                                                                                  
 block6e_bn (BatchNormalization  (None, 2, 2, 1248)  4992        ['block6e_dwconv2[0][0]']        
 )                                                                                                
                                                                                                  
 block6e_activation (Activation  (None, 2, 2, 1248)  0           ['block6e_bn[0][0]']             
 )                                                                                                
                                                                                                  
 block6e_se_squeeze (GlobalAver  (None, 1248)        0           ['block6e_activation[0][0]']     
 agePooling2D)                                                                                    
                                                                                                  
 block6e_se_reshape (Reshape)   (None, 1, 1, 1248)   0           ['block6e_se_squeeze[0][0]']     
                                                                                                  
 block6e_se_reduce (Conv2D)     (None, 1, 1, 52)     64948       ['block6e_se_reshape[0][0]']     
                                                                                                  
 block6e_se_expand (Conv2D)     (None, 1, 1, 1248)   66144       ['block6e_se_reduce[0][0]']      
                                                                                                  
 block6e_se_excite (Multiply)   (None, 2, 2, 1248)   0           ['block6e_activation[0][0]',     
                                                                  'block6e_se_expand[0][0]']      
                                                                                                  
 block6e_project_conv (Conv2D)  (None, 2, 2, 208)    259584      ['block6e_se_excite[0][0]']      
                                                                                                  
 block6e_project_bn (BatchNorma  (None, 2, 2, 208)   832         ['block6e_project_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 block6e_drop (Dropout)         (None, 2, 2, 208)    0           ['block6e_project_bn[0][0]']     
                                                                                                  
 block6e_add (Add)              (None, 2, 2, 208)    0           ['block6e_drop[0][0]',           
                                                                  'block6d_add[0][0]']            
                                                                                                  
 block6f_expand_conv (Conv2D)   (None, 2, 2, 1248)   259584      ['block6e_add[0][0]']            
                                                                                                  
 block6f_expand_bn (BatchNormal  (None, 2, 2, 1248)  4992        ['block6f_expand_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 block6f_expand_activation (Act  (None, 2, 2, 1248)  0           ['block6f_expand_bn[0][0]']      
 ivation)                                                                                         
                                                                                                  
 block6f_dwconv2 (DepthwiseConv  (None, 2, 2, 1248)  11232       ['block6f_expand_activation[0][0]
 2D)                                                             ']                               
                                                                                                  
 block6f_bn (BatchNormalization  (None, 2, 2, 1248)  4992        ['block6f_dwconv2[0][0]']        
 )                                                                                                
                                                                                                  
 block6f_activation (Activation  (None, 2, 2, 1248)  0           ['block6f_bn[0][0]']             
 )                                                                                                
                                                                                                  
 block6f_se_squeeze (GlobalAver  (None, 1248)        0           ['block6f_activation[0][0]']     
 agePooling2D)                                                                                    
                                                                                                  
 block6f_se_reshape (Reshape)   (None, 1, 1, 1248)   0           ['block6f_se_squeeze[0][0]']     
                                                                                                  
 block6f_se_reduce (Conv2D)     (None, 1, 1, 52)     64948       ['block6f_se_reshape[0][0]']     
                                                                                                  
 block6f_se_expand (Conv2D)     (None, 1, 1, 1248)   66144       ['block6f_se_reduce[0][0]']      
                                                                                                  
 block6f_se_excite (Multiply)   (None, 2, 2, 1248)   0           ['block6f_activation[0][0]',     
                                                                  'block6f_se_expand[0][0]']      
                                                                                                  
 block6f_project_conv (Conv2D)  (None, 2, 2, 208)    259584      ['block6f_se_excite[0][0]']      
                                                                                                  
 block6f_project_bn (BatchNorma  (None, 2, 2, 208)   832         ['block6f_project_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 block6f_drop (Dropout)         (None, 2, 2, 208)    0           ['block6f_project_bn[0][0]']     
                                                                                                  
 block6f_add (Add)              (None, 2, 2, 208)    0           ['block6f_drop[0][0]',           
                                                                  'block6e_add[0][0]']            
                                                                                                  
 block6g_expand_conv (Conv2D)   (None, 2, 2, 1248)   259584      ['block6f_add[0][0]']            
                                                                                                  
 block6g_expand_bn (BatchNormal  (None, 2, 2, 1248)  4992        ['block6g_expand_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 block6g_expand_activation (Act  (None, 2, 2, 1248)  0           ['block6g_expand_bn[0][0]']      
 ivation)                                                                                         
                                                                                                  
 block6g_dwconv2 (DepthwiseConv  (None, 2, 2, 1248)  11232       ['block6g_expand_activation[0][0]
 2D)                                                             ']                               
                                                                                                  
 block6g_bn (BatchNormalization  (None, 2, 2, 1248)  4992        ['block6g_dwconv2[0][0]']        
 )                                                                                                
                                                                                                  
 block6g_activation (Activation  (None, 2, 2, 1248)  0           ['block6g_bn[0][0]']             
 )                                                                                                
                                                                                                  
 block6g_se_squeeze (GlobalAver  (None, 1248)        0           ['block6g_activation[0][0]']     
 agePooling2D)                                                                                    
                                                                                                  
 block6g_se_reshape (Reshape)   (None, 1, 1, 1248)   0           ['block6g_se_squeeze[0][0]']     
                                                                                                  
 block6g_se_reduce (Conv2D)     (None, 1, 1, 52)     64948       ['block6g_se_reshape[0][0]']     
                                                                                                  
 block6g_se_expand (Conv2D)     (None, 1, 1, 1248)   66144       ['block6g_se_reduce[0][0]']      
                                                                                                  
 block6g_se_excite (Multiply)   (None, 2, 2, 1248)   0           ['block6g_activation[0][0]',     
                                                                  'block6g_se_expand[0][0]']      
                                                                                                  
 block6g_project_conv (Conv2D)  (None, 2, 2, 208)    259584      ['block6g_se_excite[0][0]']      
                                                                                                  
 block6g_project_bn (BatchNorma  (None, 2, 2, 208)   832         ['block6g_project_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 block6g_drop (Dropout)         (None, 2, 2, 208)    0           ['block6g_project_bn[0][0]']     
                                                                                                  
 block6g_add (Add)              (None, 2, 2, 208)    0           ['block6g_drop[0][0]',           
                                                                  'block6f_add[0][0]']            
                                                                                                  
 block6h_expand_conv (Conv2D)   (None, 2, 2, 1248)   259584      ['block6g_add[0][0]']            
                                                                                                  
 block6h_expand_bn (BatchNormal  (None, 2, 2, 1248)  4992        ['block6h_expand_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 block6h_expand_activation (Act  (None, 2, 2, 1248)  0           ['block6h_expand_bn[0][0]']      
 ivation)                                                                                         
                                                                                                  
 block6h_dwconv2 (DepthwiseConv  (None, 2, 2, 1248)  11232       ['block6h_expand_activation[0][0]
 2D)                                                             ']                               
                                                                                                  
 block6h_bn (BatchNormalization  (None, 2, 2, 1248)  4992        ['block6h_dwconv2[0][0]']        
 )                                                                                                
                                                                                                  
 block6h_activation (Activation  (None, 2, 2, 1248)  0           ['block6h_bn[0][0]']             
 )                                                                                                
                                                                                                  
 block6h_se_squeeze (GlobalAver  (None, 1248)        0           ['block6h_activation[0][0]']     
 agePooling2D)                                                                                    
                                                                                                  
 block6h_se_reshape (Reshape)   (None, 1, 1, 1248)   0           ['block6h_se_squeeze[0][0]']     
                                                                                                  
 block6h_se_reduce (Conv2D)     (None, 1, 1, 52)     64948       ['block6h_se_reshape[0][0]']     
                                                                                                  
 block6h_se_expand (Conv2D)     (None, 1, 1, 1248)   66144       ['block6h_se_reduce[0][0]']      
                                                                                                  
 block6h_se_excite (Multiply)   (None, 2, 2, 1248)   0           ['block6h_activation[0][0]',     
                                                                  'block6h_se_expand[0][0]']      
                                                                                                  
 block6h_project_conv (Conv2D)  (None, 2, 2, 208)    259584      ['block6h_se_excite[0][0]']      
                                                                                                  
 block6h_project_bn (BatchNorma  (None, 2, 2, 208)   832         ['block6h_project_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 block6h_drop (Dropout)         (None, 2, 2, 208)    0           ['block6h_project_bn[0][0]']     
                                                                                                  
 block6h_add (Add)              (None, 2, 2, 208)    0           ['block6h_drop[0][0]',           
                                                                  'block6g_add[0][0]']            
                                                                                                  
 block6i_expand_conv (Conv2D)   (None, 2, 2, 1248)   259584      ['block6h_add[0][0]']            
                                                                                                  
 block6i_expand_bn (BatchNormal  (None, 2, 2, 1248)  4992        ['block6i_expand_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 block6i_expand_activation (Act  (None, 2, 2, 1248)  0           ['block6i_expand_bn[0][0]']      
 ivation)                                                                                         
                                                                                                  
 block6i_dwconv2 (DepthwiseConv  (None, 2, 2, 1248)  11232       ['block6i_expand_activation[0][0]
 2D)                                                             ']                               
                                                                                                  
 block6i_bn (BatchNormalization  (None, 2, 2, 1248)  4992        ['block6i_dwconv2[0][0]']        
 )                                                                                                
                                                                                                  
 block6i_activation (Activation  (None, 2, 2, 1248)  0           ['block6i_bn[0][0]']             
 )                                                                                                
                                                                                                  
 block6i_se_squeeze (GlobalAver  (None, 1248)        0           ['block6i_activation[0][0]']     
 agePooling2D)                                                                                    
                                                                                                  
 block6i_se_reshape (Reshape)   (None, 1, 1, 1248)   0           ['block6i_se_squeeze[0][0]']     
                                                                                                  
 block6i_se_reduce (Conv2D)     (None, 1, 1, 52)     64948       ['block6i_se_reshape[0][0]']     
                                                                                                  
 block6i_se_expand (Conv2D)     (None, 1, 1, 1248)   66144       ['block6i_se_reduce[0][0]']      
                                                                                                  
 block6i_se_excite (Multiply)   (None, 2, 2, 1248)   0           ['block6i_activation[0][0]',     
                                                                  'block6i_se_expand[0][0]']      
                                                                                                  
 block6i_project_conv (Conv2D)  (None, 2, 2, 208)    259584      ['block6i_se_excite[0][0]']      
                                                                                                  
 block6i_project_bn (BatchNorma  (None, 2, 2, 208)   832         ['block6i_project_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 block6i_drop (Dropout)         (None, 2, 2, 208)    0           ['block6i_project_bn[0][0]']     
                                                                                                  
 block6i_add (Add)              (None, 2, 2, 208)    0           ['block6i_drop[0][0]',           
                                                                  'block6h_add[0][0]']            
                                                                                                  
 block6j_expand_conv (Conv2D)   (None, 2, 2, 1248)   259584      ['block6i_add[0][0]']            
                                                                                                  
 block6j_expand_bn (BatchNormal  (None, 2, 2, 1248)  4992        ['block6j_expand_conv[0][0]']    
 ization)                                                                                         
                                                                                                  
 block6j_expand_activation (Act  (None, 2, 2, 1248)  0           ['block6j_expand_bn[0][0]']      
 ivation)                                                                                         
                                                                                                  
 block6j_dwconv2 (DepthwiseConv  (None, 2, 2, 1248)  11232       ['block6j_expand_activation[0][0]
 2D)                                                             ']                               
                                                                                                  
 block6j_bn (BatchNormalization  (None, 2, 2, 1248)  4992        ['block6j_dwconv2[0][0]']        
 )                                                                                                
                                                                                                  
 block6j_activation (Activation  (None, 2, 2, 1248)  0           ['block6j_bn[0][0]']             
 )                                                                                                
                                                                                                  
 block6j_se_squeeze (GlobalAver  (None, 1248)        0           ['block6j_activation[0][0]']     
 agePooling2D)                                                                                    
                                                                                                  
 block6j_se_reshape (Reshape)   (None, 1, 1, 1248)   0           ['block6j_se_squeeze[0][0]']     
                                                                                                  
 block6j_se_reduce (Conv2D)     (None, 1, 1, 52)     64948       ['block6j_se_reshape[0][0]']     
                                                                                                  
 block6j_se_expand (Conv2D)     (None, 1, 1, 1248)   66144       ['block6j_se_reduce[0][0]']      
                                                                                                  
 block6j_se_excite (Multiply)   (None, 2, 2, 1248)   0           ['block6j_activation[0][0]',     
                                                                  'block6j_se_expand[0][0]']      
                                                                                                  
 block6j_project_conv (Conv2D)  (None, 2, 2, 208)    259584      ['block6j_se_excite[0][0]']      
                                                                                                  
 block6j_project_bn (BatchNorma  (None, 2, 2, 208)   832         ['block6j_project_conv[0][0]']   
 lization)                                                                                        
                                                                                                  
 block6j_drop (Dropout)         (None, 2, 2, 208)    0           ['block6j_project_bn[0][0]']     
                                                                                                  
 block6j_add (Add)              (None, 2, 2, 208)    0           ['block6j_drop[0][0]',           
                                                                  'block6i_add[0][0]']            
                                                                                                  
 top_conv (Conv2D)              (None, 2, 2, 1408)   292864      ['block6j_add[0][0]']            
                                                                                                  
 top_bn (BatchNormalization)    (None, 2, 2, 1408)   5632        ['top_conv[0][0]']               
                                                                                                  
 top_activation (Activation)    (None, 2, 2, 1408)   0           ['top_bn[0][0]']                 
                                                                                                  
==================================================================================================
Total params: 8,769,374
Trainable params: 8,687,086
Non-trainable params: 82,288
__________________________________________________________________________________________________
In [63]:
# Model 5: EfficientNet
def model_builder_efficientnet(input_shape, tune=0):
  """Transfer approach with the EfficientNet Model
  """
  efficientnet_model = ap.EfficientNetV2B2(include_top=False, weights="imagenet", input_shape=input_shape)
  # Making all the layers of the VGG model non-trainable
  if tune > 0:
    for layer in efficientnet_model.layers[:-tune]:
        layer.trainable = False
  else:
    for layer in efficientnet_model.layers:
        layer.trainable = False

  model = Sequential([
      efficientnet_model,
      Flatten(),
      Dense(256, activation='relu'),
      Dense(128, activation='relu'),
      Dropout(0.3),
      Dense(64, activation='relu'),
      BatchNormalization(),
      Dense(4, activation='softmax')
  ])

  # Compile model
  optimizer = Adam(learning_rate = 0.001)
  model.compile(optimizer=optimizer, loss='categorical_crossentropy', metrics=['accuracy'])

  # Generating the summary of the model
  print(model.summary())
  return model
In [64]:
batch_size = 32
model_efficientnet, metric_efficientnet, history_efficientnet = model_building_and_evaluating_process(
    model_builder_efficientnet, f'Model 5: EfficientNet ({batch_size})', batch_size=batch_size, epochs=20, include_grayscale=False
)
---------------------------------------------
Model 5: EfficientNet (32) - RGB
---------------------------------------------
Initializing TF Session and Seed

Data with color_mode=rgb:
Found 15109 images belonging to 4 classes.
Found 127 images belonging to 4 classes.
Found 4977 images belonging to 4 classes.



Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 efficientnetv2-b2 (Function  (None, 2, 2, 1408)       8769374   
 al)                                                             
                                                                 
 flatten (Flatten)           (None, 5632)              0         
                                                                 
 dense (Dense)               (None, 256)               1442048   
                                                                 
 dense_1 (Dense)             (None, 128)               32896     
                                                                 
 dropout (Dropout)           (None, 128)               0         
                                                                 
 dense_2 (Dense)             (None, 64)                8256      
                                                                 
 batch_normalization (BatchN  (None, 64)               256       
 ormalization)                                                   
                                                                 
 dense_3 (Dense)             (None, 4)                 260       
                                                                 
=================================================================
Total params: 10,253,090
Trainable params: 1,483,588
Non-trainable params: 8,769,502
_________________________________________________________________
None



Epoch 1/20
473/473 [==============================] - 51s 84ms/step - loss: 1.4558 - accuracy: 0.2583 - val_loss: 1.4238 - val_accuracy: 0.1601 - lr: 0.0010
Epoch 2/20
473/473 [==============================] - 38s 79ms/step - loss: 1.4049 - accuracy: 0.2586 - val_loss: 1.3468 - val_accuracy: 0.3667 - lr: 0.0010
Epoch 3/20
473/473 [==============================] - 36s 76ms/step - loss: 1.3985 - accuracy: 0.2543 - val_loss: 1.3631 - val_accuracy: 0.3667 - lr: 0.0010
Epoch 4/20
473/473 [==============================] - 36s 77ms/step - loss: 1.3937 - accuracy: 0.2547 - val_loss: 1.3806 - val_accuracy: 0.2443 - lr: 0.0010
Epoch 5/20
473/473 [==============================] - 36s 77ms/step - loss: 1.3930 - accuracy: 0.2603 - val_loss: 1.3744 - val_accuracy: 0.2443 - lr: 0.0010
Epoch 6/20
473/473 [==============================] - 36s 77ms/step - loss: 1.3860 - accuracy: 0.2605 - val_loss: 1.3871 - val_accuracy: 0.2289 - lr: 2.0000e-04
Epoch 7/20
473/473 [==============================] - 38s 79ms/step - loss: 1.3839 - accuracy: 0.2639 - val_loss: 1.3834 - val_accuracy: 0.3679 - lr: 2.0000e-04



1/1 [==============================] - 0s 74ms/step - loss: 1.4044 - accuracy: 0.3125
1/1 [==============================] - 2s 2s/step
Actual    : [0 1 3 0 1 3 2 1 0 2 0 3 2 2 3 0 3 2 0 2 1 3 2 2 0 0 3 0 1 2 0 2]
Prediction: [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]



              precision    recall  f1-score   support

       happy       0.31      1.00      0.48        10
     neutral       0.00      0.00      0.00         5
         sad       0.00      0.00      0.00        10
    surprise       0.00      0.00      0.00         7

    accuracy                           0.31        32
   macro avg       0.08      0.25      0.12        32
weighted avg       0.10      0.31      0.15        32





In [65]:
# Let's try again tuning some of the last layers in the transfer model
batch_size = 32
model, results, history = model_building_and_evaluating_process(model_builder_efficientnet, f'Model 5: Tuned EfficientNet ({batch_size})',
                                                                batch_size=batch_size, epochs=20, tune=7, include_grayscale=False)
model_efficientnet.update(model)
metric_efficientnet = pd.concat([metric_efficientnet, results])
history_efficientnet.update(history)
---------------------------------------------
Model 5: Tuned EfficientNet (32) - RGB
---------------------------------------------
Initializing TF Session and Seed

Data with color_mode=rgb:
Found 15109 images belonging to 4 classes.
Found 127 images belonging to 4 classes.
Found 4977 images belonging to 4 classes.



Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 efficientnetv2-b2 (Function  (None, 2, 2, 1408)       8769374   
 al)                                                             
                                                                 
 flatten (Flatten)           (None, 5632)              0         
                                                                 
 dense (Dense)               (None, 256)               1442048   
                                                                 
 dense_1 (Dense)             (None, 128)               32896     
                                                                 
 dropout (Dropout)           (None, 128)               0         
                                                                 
 dense_2 (Dense)             (None, 64)                8256      
                                                                 
 batch_normalization (BatchN  (None, 64)               256       
 ormalization)                                                   
                                                                 
 dense_3 (Dense)             (None, 4)                 260       
                                                                 
=================================================================
Total params: 10,253,090
Trainable params: 2,039,268
Non-trainable params: 8,213,822
_________________________________________________________________
None



Epoch 1/20
473/473 [==============================] - 54s 90ms/step - loss: 1.4554 - accuracy: 0.2542 - val_loss: 1.3559 - val_accuracy: 0.3667 - lr: 0.0010
Epoch 2/20
473/473 [==============================] - 38s 80ms/step - loss: 1.3994 - accuracy: 0.2585 - val_loss: 1.3627 - val_accuracy: 0.3667 - lr: 0.0010
Epoch 3/20
473/473 [==============================] - 38s 81ms/step - loss: 1.3935 - accuracy: 0.2574 - val_loss: 1.3700 - val_accuracy: 0.3667 - lr: 0.0010
Epoch 4/20
473/473 [==============================] - 39s 83ms/step - loss: 1.3933 - accuracy: 0.2640 - val_loss: 1.3663 - val_accuracy: 0.3671 - lr: 0.0010
Epoch 5/20
473/473 [==============================] - 38s 81ms/step - loss: 1.3865 - accuracy: 0.2595 - val_loss: 1.3703 - val_accuracy: 0.2289 - lr: 2.0000e-04
Epoch 6/20
473/473 [==============================] - 39s 82ms/step - loss: 1.3851 - accuracy: 0.2637 - val_loss: 1.3658 - val_accuracy: 0.3667 - lr: 2.0000e-04



1/1 [==============================] - 0s 80ms/step - loss: 1.3944 - accuracy: 0.3125
1/1 [==============================] - 2s 2s/step
Actual    : [0 1 3 0 1 3 2 1 0 2 0 3 2 2 3 0 3 2 0 2 1 3 2 2 0 0 3 0 1 2 0 2]
Prediction: [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]



              precision    recall  f1-score   support

       happy       0.31      1.00      0.48        10
     neutral       0.00      0.00      0.00         5
         sad       0.00      0.00      0.00        10
    surprise       0.00      0.00      0.00         7

    accuracy                           0.31        32
   macro avg       0.08      0.25      0.12        32
weighted avg       0.10      0.31      0.15        32





In [66]:
model_name = 'Model 5: EfficientNet'
save_object(history_efficientnet, f'{MODEL_DIR}/{model_name} - history.pkl')
save_object(metric_efficientnet, f'{MODEL_DIR}/{model_name} - metrics.pkl')
save_object(model_efficientnet, f'{MODEL_DIR}/{model_name} - models.pkl')
In [67]:
loss_and_accuracy_comparisson(history_efficientnet, data_type='Training Set')
loss_and_accuracy_comparisson(history_efficientnet, data_type='Validation Set')
In [68]:
metric_efficientnet.style.highlight_max(color = "lightgreen", axis = 0)
Out[68]:
  Accuracy Train Accuracy Val Accuracy Test
Model 5: EfficientNet (32) - RGB 0.263949 0.367892 0.312500
Model 5: Tuned EfficientNet (32) - RGB 0.264015 0.367089 0.312500

Note: You can even go back and build your own architecture on top of the VGG16 Transfer layer and see if you can improve the performance.

⏩ Observations and Insights:

  • The EfficientNet is a type of Neural Network architecture that uses compound Scaling to enable better performance on the ImageNet classification task.
  • For the current approach in model no. 5, it was taken from EfficientNet the convolutional layers block (561 layers), and joined to the classification layers composed mainly of Dense layers, the idea was to verify if this approach improves the results obtained with the previous models.
  • On the first try, it was turned the trainable property of all the convolutional layers (extracted from EfficientNet), to False, so it was ensured that the weight learned in EfficientNet will be applied to the new problem. This provided 36.8% of accuracy, not good at all.
  • In the second try, the trainable property = False was not applied to all convolutional layers and when the model was compiled it allows the backpropagation to update the last 7 pre-trained layers. The 7 last convolutional layers were trained with the data to help the model learn specific features from the data in this specific image classification problem. This second try achieved similar results: 36.7% of accuracy. No improvement at all, if we review the predicted class in the testing validation, all values are the same, meaning that this model could not reveal the features and learn to have a good performance.
  • The learning curve reflects underfitting, the model is not learning the features of the dataset. Increasing the layers at the end of the convolutional block to retrain is advisable to try to improve its performance.
  • The best model "Model 5: EfficientNet (32) - RGB" achieved 36.8% accuracy.

⏩ Recommendations:

  • Increase the number of convolutional layers to retrain with the data, by increasing the "tune" parameter in the created method "model_builder_efficientnet".
  • Explore other image augmentation techniques to provide more data to the model.
  • A review of the training dataset is necessary to confirm data are good enough.

Think About It:

  • What is your overall performance of these Transfer Learning Architectures? Can we draw a comparison of these models' performances. Are we satisfied with the accuracies that we have received?
  • Do you think our issue lies with 'rgb' color_mode?
In [69]:
# Getting back the history and metrics result during trainning
n = 'Model 3: VGG16'
with open(f'{MODEL_DIR}/{n} - history.pkl', 'rb') as f: history_vgg16 = pickle.load(f)
with open(f'{MODEL_DIR}/{n} - metrics.pkl', 'rb') as f: metric_vgg16 = pickle.load(f)

n = 'Model 4: Resnet V2'
with open(f'{MODEL_DIR}/{n} - history.pkl', 'rb') as f: history_resnet = pickle.load(f)
with open(f'{MODEL_DIR}/{n} - metrics.pkl', 'rb') as f: metric_resnet = pickle.load(f)

n = 'Model 5: EfficientNet'
with open(f'{MODEL_DIR}/{n} - history.pkl', 'rb') as f: history_efficientnet = pickle.load(f)
with open(f'{MODEL_DIR}/{n} - metrics.pkl', 'rb') as f: metric_efficientnet = pickle.load(f)
In [70]:
history_transfer_models = {
    'Model 3: Tuned VGG16 (32) - RGB': history_vgg16['Model 3: Tuned VGG16 (32) - RGB'],
    'Model 4: Tuned Resnet V2 (32) - RGB': history_resnet['Model 4: Tuned Resnet V2 (32) - RGB'],
    'Model 5: Tuned EfficientNet (32) - RGB': history_efficientnet['Model 5: Tuned EfficientNet (32) - RGB']
}
loss_and_accuracy_comparisson(history_transfer_models, data_type='Training Set')
loss_and_accuracy_comparisson(history_transfer_models, data_type='Validation Set')
In [71]:
metric_transfer_models = pd.concat([
    metric_vgg16.loc['Model 3: Tuned VGG16 (32) - RGB':],
    metric_resnet.loc['Model 4: Tuned Resnet V2 (32) - RGB':],
    metric_efficientnet.loc['Model 5: Tuned EfficientNet (32) - RGB':]
])
metric_transfer_models
Out[71]:
Accuracy Train Accuracy Val Accuracy Test
Model 3: Tuned VGG16 (32) - RGB 0.784367 0.712879 0.8125
Model 4: Tuned Resnet V2 (32) - RGB 0.513667 0.506731 0.3750
Model 5: Tuned EfficientNet (32) - RGB 0.264015 0.367089 0.3125
In [72]:
_ = metric_transfer_models.plot(kind='bar', figsize=(10,3))
plt.xticks(rotation=0, fontsize=8)
plt.yticks(fontsize=8)
plt.title('Best Performed Trained Transfer Models', fontsize=10)
plt.legend(fontsize=8)
plt.show()
In [73]:
pd.concat([metric_vgg16, metric_resnet, metric_efficientnet]).plot(kind='bar', figsize=(15,3))
plt.xticks(rotation=0, fontsize=8)
plt.yticks(fontsize=8)
plt.title('Trained Transfer Models', fontsize=10)
plt.legend(fontsize=8)
plt.show()

⏩ Observations and Insights:

  • Making the comparison across all trained transfer models, we can conclude that tuning helped the model to improve. This tuning means allowing backpropagation to retrain the n-last layers of the convolutional block obtained from the base model (VGG16, Resnet V2, EfficientNet). This is because we allow the model to find specific features related to the current problem to solve.
  • The advantages of these models are that they were trained in colored images, having the possibility to capture features from the color information so our transfer models could use those weights learned with images encoded in 3 channels (RGB) as our current dataset. Remember they are in gray color, but they are encoded in 3 channels.
  • The bad performance of some of these trained transfer models could be related to the tuning, maybe this "tune" parameter could be increased to have more final layers of the convolutional block being retrained. The last 2 base models: ResNet and EfficientNet obtained the lower accuracy (48.3% and 33% respectively) but their tunning was very low in comparison to the number of layers they have in the convolutional block (for ResNet, tune=5 having 546 layers, for EfficientNet, tune=7 having 561 layers).

Now that we have tried multiple pre-trained models, let's build a complex CNN architecture and see if we can get better performance.

Building a Complex Neural Network Architecture¶

In this section, we will build a more complex Convolutional Neural Network Model that has close to as many parameters as we had in our Transfer Learning Models. However, we will have only 1 input channel for our input images.

Creating our Data Loaders¶

In this section, we are creating data loaders which we will use as inputs to the more Complicated Convolutional Neural Network. We will go ahead with color_mode = 'grayscale'.

In [74]:
train_generator_gray, _, _ = create_data_loaders(IMG_DIR, 'grayscale')

# Taking a look at some examples of our augmented training data with color_mode='rgb'.
images, labels = next(train_generator_gray)
fig, axes = plt.subplots(1, 8, figsize = (10, 2))
for (image, label, ax) in zip(images, labels, axes.flatten()):
  ax.imshow(image, cmap='gray')
  ax.set_title(FACE_EXPRESSIONS[list(label).index(1)], fontsize=8)
  ax.axis('off')
plt.show()
Data with color_mode=grayscale:
Found 15109 images belonging to 4 classes.
Found 127 images belonging to 4 classes.
Found 4977 images belonging to 4 classes.

Model 6: Complex Neural Network¶


  • Try building a layer with 5 Convolutional Blocks and see if performance increases.
In [75]:
# Model 6: Complex Neural Network
def model_builder_complex_cnn(input_shape):
  """Creating a Complex Neural Network.
  """
  model = Sequential([
      Conv2D(filters=64, kernel_size=2, activation='relu', padding='same', input_shape=input_shape),
      BatchNormalization(),
      LeakyReLU(0.1),
      MaxPooling2D(pool_size=2),
      Dropout(0.2),

      Conv2D(filters=128, kernel_size=2, activation='relu', padding='same'),
      BatchNormalization(),
      LeakyReLU(0.1),
      MaxPooling2D(pool_size=2),
      Dropout(0.2),

      Conv2D(filters=512, kernel_size=2, activation='relu', padding='same'),
      BatchNormalization(),
      LeakyReLU(0.1),
      MaxPooling2D(pool_size=2),
      Dropout(0.2),

      Conv2D(filters=512, kernel_size=2, activation='relu', padding='same'),
      BatchNormalization(),
      LeakyReLU(0.1),
      MaxPooling2D(pool_size=2),
      Dropout(0.2),

      Conv2D(filters=128, kernel_size=2, activation='relu', padding='same'),
      BatchNormalization(),
      LeakyReLU(0.1),
      MaxPooling2D(pool_size=2),
      Dropout(0.2),

      Flatten(),

      Dense(256, activation='relu'),
      BatchNormalization(),
      Dropout(0.4),

      Dense(512, activation='relu'),
      BatchNormalization(),
      Dropout(0.4),

      Dense(4, activation='softmax')
  ])

  # Compile model
  optimizer = Adam(learning_rate = 0.01)        # Using SGD Optimizer
  model.compile(optimizer=optimizer, loss='categorical_crossentropy', metrics=['accuracy'])

  # Generating the summary of the model
  print(model.summary())
  return model
In [76]:
batch_size = 32
model_complex_cnn, metric_complex_cnn, history_complex_cnn = model_building_and_evaluating_process(
    model_builder_complex_cnn, f'Model 6: Complex CNN ({batch_size})', batch_size=batch_size, epochs=20, include_rgb=False
)
---------------------------------------------
Model 6: Complex CNN (32) - GRAY
---------------------------------------------
Initializing TF Session and Seed

Data with color_mode=grayscale:
Found 15109 images belonging to 4 classes.
Found 127 images belonging to 4 classes.
Found 4977 images belonging to 4 classes.



Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d (Conv2D)             (None, 48, 48, 64)        320       
                                                                 
 batch_normalization (BatchN  (None, 48, 48, 64)       256       
 ormalization)                                                   
                                                                 
 leaky_re_lu (LeakyReLU)     (None, 48, 48, 64)        0         
                                                                 
 max_pooling2d (MaxPooling2D  (None, 24, 24, 64)       0         
 )                                                               
                                                                 
 dropout (Dropout)           (None, 24, 24, 64)        0         
                                                                 
 conv2d_1 (Conv2D)           (None, 24, 24, 128)       32896     
                                                                 
 batch_normalization_1 (Batc  (None, 24, 24, 128)      512       
 hNormalization)                                                 
                                                                 
 leaky_re_lu_1 (LeakyReLU)   (None, 24, 24, 128)       0         
                                                                 
 max_pooling2d_1 (MaxPooling  (None, 12, 12, 128)      0         
 2D)                                                             
                                                                 
 dropout_1 (Dropout)         (None, 12, 12, 128)       0         
                                                                 
 conv2d_2 (Conv2D)           (None, 12, 12, 512)       262656    
                                                                 
 batch_normalization_2 (Batc  (None, 12, 12, 512)      2048      
 hNormalization)                                                 
                                                                 
 leaky_re_lu_2 (LeakyReLU)   (None, 12, 12, 512)       0         
                                                                 
 max_pooling2d_2 (MaxPooling  (None, 6, 6, 512)        0         
 2D)                                                             
                                                                 
 dropout_2 (Dropout)         (None, 6, 6, 512)         0         
                                                                 
 conv2d_3 (Conv2D)           (None, 6, 6, 512)         1049088   
                                                                 
 batch_normalization_3 (Batc  (None, 6, 6, 512)        2048      
 hNormalization)                                                 
                                                                 
 leaky_re_lu_3 (LeakyReLU)   (None, 6, 6, 512)         0         
                                                                 
 max_pooling2d_3 (MaxPooling  (None, 3, 3, 512)        0         
 2D)                                                             
                                                                 
 dropout_3 (Dropout)         (None, 3, 3, 512)         0         
                                                                 
 conv2d_4 (Conv2D)           (None, 3, 3, 128)         262272    
                                                                 
 batch_normalization_4 (Batc  (None, 3, 3, 128)        512       
 hNormalization)                                                 
                                                                 
 leaky_re_lu_4 (LeakyReLU)   (None, 3, 3, 128)         0         
                                                                 
 max_pooling2d_4 (MaxPooling  (None, 1, 1, 128)        0         
 2D)                                                             
                                                                 
 dropout_4 (Dropout)         (None, 1, 1, 128)         0         
                                                                 
 flatten (Flatten)           (None, 128)               0         
                                                                 
 dense (Dense)               (None, 256)               33024     
                                                                 
 batch_normalization_5 (Batc  (None, 256)              1024      
 hNormalization)                                                 
                                                                 
 dropout_5 (Dropout)         (None, 256)               0         
                                                                 
 dense_1 (Dense)             (None, 512)               131584    
                                                                 
 batch_normalization_6 (Batc  (None, 512)              2048      
 hNormalization)                                                 
                                                                 
 dropout_6 (Dropout)         (None, 512)               0         
                                                                 
 dense_2 (Dense)             (None, 4)                 2052      
                                                                 
=================================================================
Total params: 1,782,340
Trainable params: 1,778,116
Non-trainable params: 4,224
_________________________________________________________________
None



Epoch 1/20
473/473 [==============================] - 44s 83ms/step - loss: 1.7116 - accuracy: 0.2803 - val_loss: 1.5907 - val_accuracy: 0.2536 - lr: 0.0100
Epoch 2/20
473/473 [==============================] - 39s 82ms/step - loss: 1.3762 - accuracy: 0.3332 - val_loss: 1.2800 - val_accuracy: 0.3940 - lr: 0.0100
Epoch 3/20
473/473 [==============================] - 39s 82ms/step - loss: 1.2049 - accuracy: 0.4303 - val_loss: 1.2257 - val_accuracy: 0.4577 - lr: 0.0100
Epoch 4/20
473/473 [==============================] - 39s 82ms/step - loss: 1.0813 - accuracy: 0.5174 - val_loss: 1.0792 - val_accuracy: 0.5248 - lr: 0.0100
Epoch 5/20
473/473 [==============================] - 39s 82ms/step - loss: 1.0102 - accuracy: 0.5609 - val_loss: 0.9836 - val_accuracy: 0.5817 - lr: 0.0100
Epoch 6/20
473/473 [==============================] - 38s 81ms/step - loss: 0.9574 - accuracy: 0.5900 - val_loss: 1.1192 - val_accuracy: 0.5357 - lr: 0.0100
Epoch 7/20
473/473 [==============================] - 38s 80ms/step - loss: 0.9274 - accuracy: 0.6116 - val_loss: 1.1068 - val_accuracy: 0.5043 - lr: 0.0100
Epoch 8/20
473/473 [==============================] - 39s 83ms/step - loss: 0.9071 - accuracy: 0.6203 - val_loss: 0.8265 - val_accuracy: 0.6536 - lr: 0.0100
Epoch 9/20
473/473 [==============================] - 38s 81ms/step - loss: 0.8800 - accuracy: 0.6366 - val_loss: 0.8600 - val_accuracy: 0.6564 - lr: 0.0100
Epoch 10/20
473/473 [==============================] - 38s 81ms/step - loss: 0.8668 - accuracy: 0.6406 - val_loss: 0.8620 - val_accuracy: 0.6323 - lr: 0.0100
Epoch 11/20
473/473 [==============================] - 38s 80ms/step - loss: 0.8432 - accuracy: 0.6469 - val_loss: 3.5923 - val_accuracy: 0.5337 - lr: 0.0100
Epoch 12/20
473/473 [==============================] - 39s 82ms/step - loss: 0.7678 - accuracy: 0.6855 - val_loss: 0.6736 - val_accuracy: 0.7281 - lr: 0.0020
Epoch 13/20
473/473 [==============================] - 38s 81ms/step - loss: 0.7385 - accuracy: 0.7013 - val_loss: 0.6852 - val_accuracy: 0.7231 - lr: 0.0020
Epoch 14/20
473/473 [==============================] - 38s 81ms/step - loss: 0.7189 - accuracy: 0.7120 - val_loss: 0.6652 - val_accuracy: 0.7275 - lr: 0.0020
Epoch 15/20
473/473 [==============================] - 38s 81ms/step - loss: 0.7087 - accuracy: 0.7169 - val_loss: 0.6839 - val_accuracy: 0.7241 - lr: 0.0020
Epoch 16/20
473/473 [==============================] - 39s 82ms/step - loss: 0.6888 - accuracy: 0.7221 - val_loss: 0.6374 - val_accuracy: 0.7434 - lr: 0.0020
Epoch 17/20
473/473 [==============================] - 39s 83ms/step - loss: 0.6760 - accuracy: 0.7274 - val_loss: 0.6516 - val_accuracy: 0.7326 - lr: 0.0020
Epoch 18/20
473/473 [==============================] - 40s 84ms/step - loss: 0.6656 - accuracy: 0.7339 - val_loss: 0.6614 - val_accuracy: 0.7296 - lr: 0.0020
Epoch 19/20
473/473 [==============================] - 40s 84ms/step - loss: 0.6519 - accuracy: 0.7387 - val_loss: 0.6358 - val_accuracy: 0.7472 - lr: 0.0020
Epoch 20/20
473/473 [==============================] - 38s 81ms/step - loss: 0.6468 - accuracy: 0.7395 - val_loss: 0.7484 - val_accuracy: 0.6908 - lr: 0.0020



1/1 [==============================] - 0s 38ms/step - loss: 0.7757 - accuracy: 0.7500
1/1 [==============================] - 0s 199ms/step
Actual    : [0 1 3 0 1 3 2 1 0 2 0 3 2 2 3 0 3 2 0 2 1 3 2 2 0 0 3 0 1 2 0 2]
Prediction: [0 2 3 0 2 3 2 1 0 2 1 3 2 0 3 0 3 2 0 2 2 2 2 2 0 2 1 0 1 2 0 2]



              precision    recall  f1-score   support

       happy       0.89      0.80      0.84        10
     neutral       0.50      0.40      0.44         5
         sad       0.64      0.90      0.75        10
    surprise       1.00      0.71      0.83         7

    accuracy                           0.75        32
   macro avg       0.76      0.70      0.72        32
weighted avg       0.78      0.75      0.75        32





In [77]:
batch_size = 64
model, results, history = model_building_and_evaluating_process(model_builder_complex_cnn, f'Model 6: Complex CNN ({batch_size})',
                                                                batch_size=batch_size, epochs=20, include_rgb=False)
model_complex_cnn.update(model)
metric_complex_cnn = pd.concat([metric_complex_cnn, results])
history_complex_cnn.update(history)
---------------------------------------------
Model 6: Complex CNN (64) - GRAY
---------------------------------------------
Initializing TF Session and Seed

Data with color_mode=grayscale:
Found 15109 images belonging to 4 classes.
Found 127 images belonging to 4 classes.
Found 4977 images belonging to 4 classes.



Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d (Conv2D)             (None, 48, 48, 64)        320       
                                                                 
 batch_normalization (BatchN  (None, 48, 48, 64)       256       
 ormalization)                                                   
                                                                 
 leaky_re_lu (LeakyReLU)     (None, 48, 48, 64)        0         
                                                                 
 max_pooling2d (MaxPooling2D  (None, 24, 24, 64)       0         
 )                                                               
                                                                 
 dropout (Dropout)           (None, 24, 24, 64)        0         
                                                                 
 conv2d_1 (Conv2D)           (None, 24, 24, 128)       32896     
                                                                 
 batch_normalization_1 (Batc  (None, 24, 24, 128)      512       
 hNormalization)                                                 
                                                                 
 leaky_re_lu_1 (LeakyReLU)   (None, 24, 24, 128)       0         
                                                                 
 max_pooling2d_1 (MaxPooling  (None, 12, 12, 128)      0         
 2D)                                                             
                                                                 
 dropout_1 (Dropout)         (None, 12, 12, 128)       0         
                                                                 
 conv2d_2 (Conv2D)           (None, 12, 12, 512)       262656    
                                                                 
 batch_normalization_2 (Batc  (None, 12, 12, 512)      2048      
 hNormalization)                                                 
                                                                 
 leaky_re_lu_2 (LeakyReLU)   (None, 12, 12, 512)       0         
                                                                 
 max_pooling2d_2 (MaxPooling  (None, 6, 6, 512)        0         
 2D)                                                             
                                                                 
 dropout_2 (Dropout)         (None, 6, 6, 512)         0         
                                                                 
 conv2d_3 (Conv2D)           (None, 6, 6, 512)         1049088   
                                                                 
 batch_normalization_3 (Batc  (None, 6, 6, 512)        2048      
 hNormalization)                                                 
                                                                 
 leaky_re_lu_3 (LeakyReLU)   (None, 6, 6, 512)         0         
                                                                 
 max_pooling2d_3 (MaxPooling  (None, 3, 3, 512)        0         
 2D)                                                             
                                                                 
 dropout_3 (Dropout)         (None, 3, 3, 512)         0         
                                                                 
 conv2d_4 (Conv2D)           (None, 3, 3, 128)         262272    
                                                                 
 batch_normalization_4 (Batc  (None, 3, 3, 128)        512       
 hNormalization)                                                 
                                                                 
 leaky_re_lu_4 (LeakyReLU)   (None, 3, 3, 128)         0         
                                                                 
 max_pooling2d_4 (MaxPooling  (None, 1, 1, 128)        0         
 2D)                                                             
                                                                 
 dropout_4 (Dropout)         (None, 1, 1, 128)         0         
                                                                 
 flatten (Flatten)           (None, 128)               0         
                                                                 
 dense (Dense)               (None, 256)               33024     
                                                                 
 batch_normalization_5 (Batc  (None, 256)              1024      
 hNormalization)                                                 
                                                                 
 dropout_5 (Dropout)         (None, 256)               0         
                                                                 
 dense_1 (Dense)             (None, 512)               131584    
                                                                 
 batch_normalization_6 (Batc  (None, 512)              2048      
 hNormalization)                                                 
                                                                 
 dropout_6 (Dropout)         (None, 512)               0         
                                                                 
 dense_2 (Dense)             (None, 4)                 2052      
                                                                 
=================================================================
Total params: 1,782,340
Trainable params: 1,778,116
Non-trainable params: 4,224
_________________________________________________________________
None



Epoch 1/20
237/237 [==============================] - 42s 163ms/step - loss: 1.7046 - accuracy: 0.2904 - val_loss: 1.4634 - val_accuracy: 0.2443 - lr: 0.0100
Epoch 2/20
237/237 [==============================] - 38s 159ms/step - loss: 1.4348 - accuracy: 0.3182 - val_loss: 1.3236 - val_accuracy: 0.3430 - lr: 0.0100
Epoch 3/20
237/237 [==============================] - 38s 159ms/step - loss: 1.2923 - accuracy: 0.3857 - val_loss: 1.2863 - val_accuracy: 0.3436 - lr: 0.0100
Epoch 4/20
237/237 [==============================] - 38s 158ms/step - loss: 1.1288 - accuracy: 0.4879 - val_loss: 1.1257 - val_accuracy: 0.5224 - lr: 0.0100
Epoch 5/20
237/237 [==============================] - 37s 156ms/step - loss: 1.0201 - accuracy: 0.5523 - val_loss: 1.1093 - val_accuracy: 0.4999 - lr: 0.0100
Epoch 6/20
237/237 [==============================] - 37s 157ms/step - loss: 0.9889 - accuracy: 0.5671 - val_loss: 1.4645 - val_accuracy: 0.4836 - lr: 0.0100
Epoch 7/20
237/237 [==============================] - 38s 159ms/step - loss: 0.9374 - accuracy: 0.6017 - val_loss: 0.9721 - val_accuracy: 0.5893 - lr: 0.0100
Epoch 8/20
237/237 [==============================] - 37s 156ms/step - loss: 0.9039 - accuracy: 0.6202 - val_loss: 1.1374 - val_accuracy: 0.4933 - lr: 0.0100
Epoch 9/20
237/237 [==============================] - 38s 160ms/step - loss: 0.9106 - accuracy: 0.6167 - val_loss: 0.7781 - val_accuracy: 0.6827 - lr: 0.0100
Epoch 10/20
237/237 [==============================] - 37s 156ms/step - loss: 0.8654 - accuracy: 0.6390 - val_loss: 0.9844 - val_accuracy: 0.5897 - lr: 0.0100
Epoch 11/20
237/237 [==============================] - 37s 156ms/step - loss: 0.8498 - accuracy: 0.6473 - val_loss: 0.9861 - val_accuracy: 0.5586 - lr: 0.0100
Epoch 12/20
237/237 [==============================] - 37s 157ms/step - loss: 0.8299 - accuracy: 0.6605 - val_loss: 0.8312 - val_accuracy: 0.6508 - lr: 0.0100
Epoch 13/20
237/237 [==============================] - 38s 159ms/step - loss: 0.7649 - accuracy: 0.6856 - val_loss: 0.7060 - val_accuracy: 0.7201 - lr: 0.0020
Epoch 14/20
237/237 [==============================] - 38s 159ms/step - loss: 0.7394 - accuracy: 0.6950 - val_loss: 0.6770 - val_accuracy: 0.7306 - lr: 0.0020
Epoch 15/20
237/237 [==============================] - 37s 155ms/step - loss: 0.7169 - accuracy: 0.7086 - val_loss: 0.6951 - val_accuracy: 0.7199 - lr: 0.0020
Epoch 16/20
237/237 [==============================] - 37s 156ms/step - loss: 0.7073 - accuracy: 0.7137 - val_loss: 0.7032 - val_accuracy: 0.7195 - lr: 0.0020
Epoch 17/20
237/237 [==============================] - 37s 156ms/step - loss: 0.6990 - accuracy: 0.7155 - val_loss: 0.7149 - val_accuracy: 0.7145 - lr: 0.0020
Epoch 18/20
237/237 [==============================] - 37s 158ms/step - loss: 0.6783 - accuracy: 0.7285 - val_loss: 0.6455 - val_accuracy: 0.7408 - lr: 4.0000e-04
Epoch 19/20
237/237 [==============================] - 37s 155ms/step - loss: 0.6710 - accuracy: 0.7309 - val_loss: 0.6517 - val_accuracy: 0.7392 - lr: 4.0000e-04
Epoch 20/20
237/237 [==============================] - 38s 158ms/step - loss: 0.6648 - accuracy: 0.7282 - val_loss: 0.6724 - val_accuracy: 0.7410 - lr: 4.0000e-04



1/1 [==============================] - 0s 38ms/step - loss: 0.7054 - accuracy: 0.6875
1/1 [==============================] - 0s 196ms/step
Actual    : [0 1 3 0 1 3 2 1 0 2 0 3 2 2 3 0 3 2 0 2 1 3 2 2 0 0 3 0 1 2 0 2]
Prediction: [0 1 3 0 2 3 2 1 0 1 0 3 2 0 3 0 3 2 0 2 2 2 1 1 0 2 1 0 1 1 0 2]



              precision    recall  f1-score   support

       happy       0.90      0.90      0.90        10
     neutral       0.38      0.60      0.46         5
         sad       0.56      0.50      0.53        10
    surprise       1.00      0.71      0.83         7

    accuracy                           0.69        32
   macro avg       0.71      0.68      0.68        32
weighted avg       0.73      0.69      0.70        32





In [78]:
batch_size = 128
model, results, history = model_building_and_evaluating_process(model_builder_complex_cnn, f'Model 6: Complex CNN ({batch_size})',
                                                                batch_size=batch_size, epochs=20, include_rgb=False)
model_complex_cnn.update(model)
metric_complex_cnn = pd.concat([metric_complex_cnn, results])
history_complex_cnn.update(history)
---------------------------------------------
Model 6: Complex CNN (128) - GRAY
---------------------------------------------
Initializing TF Session and Seed

Data with color_mode=grayscale:
Found 15109 images belonging to 4 classes.
Found 127 images belonging to 4 classes.
Found 4977 images belonging to 4 classes.



Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d (Conv2D)             (None, 48, 48, 64)        320       
                                                                 
 batch_normalization (BatchN  (None, 48, 48, 64)       256       
 ormalization)                                                   
                                                                 
 leaky_re_lu (LeakyReLU)     (None, 48, 48, 64)        0         
                                                                 
 max_pooling2d (MaxPooling2D  (None, 24, 24, 64)       0         
 )                                                               
                                                                 
 dropout (Dropout)           (None, 24, 24, 64)        0         
                                                                 
 conv2d_1 (Conv2D)           (None, 24, 24, 128)       32896     
                                                                 
 batch_normalization_1 (Batc  (None, 24, 24, 128)      512       
 hNormalization)                                                 
                                                                 
 leaky_re_lu_1 (LeakyReLU)   (None, 24, 24, 128)       0         
                                                                 
 max_pooling2d_1 (MaxPooling  (None, 12, 12, 128)      0         
 2D)                                                             
                                                                 
 dropout_1 (Dropout)         (None, 12, 12, 128)       0         
                                                                 
 conv2d_2 (Conv2D)           (None, 12, 12, 512)       262656    
                                                                 
 batch_normalization_2 (Batc  (None, 12, 12, 512)      2048      
 hNormalization)                                                 
                                                                 
 leaky_re_lu_2 (LeakyReLU)   (None, 12, 12, 512)       0         
                                                                 
 max_pooling2d_2 (MaxPooling  (None, 6, 6, 512)        0         
 2D)                                                             
                                                                 
 dropout_2 (Dropout)         (None, 6, 6, 512)         0         
                                                                 
 conv2d_3 (Conv2D)           (None, 6, 6, 512)         1049088   
                                                                 
 batch_normalization_3 (Batc  (None, 6, 6, 512)        2048      
 hNormalization)                                                 
                                                                 
 leaky_re_lu_3 (LeakyReLU)   (None, 6, 6, 512)         0         
                                                                 
 max_pooling2d_3 (MaxPooling  (None, 3, 3, 512)        0         
 2D)                                                             
                                                                 
 dropout_3 (Dropout)         (None, 3, 3, 512)         0         
                                                                 
 conv2d_4 (Conv2D)           (None, 3, 3, 128)         262272    
                                                                 
 batch_normalization_4 (Batc  (None, 3, 3, 128)        512       
 hNormalization)                                                 
                                                                 
 leaky_re_lu_4 (LeakyReLU)   (None, 3, 3, 128)         0         
                                                                 
 max_pooling2d_4 (MaxPooling  (None, 1, 1, 128)        0         
 2D)                                                             
                                                                 
 dropout_4 (Dropout)         (None, 1, 1, 128)         0         
                                                                 
 flatten (Flatten)           (None, 128)               0         
                                                                 
 dense (Dense)               (None, 256)               33024     
                                                                 
 batch_normalization_5 (Batc  (None, 256)              1024      
 hNormalization)                                                 
                                                                 
 dropout_5 (Dropout)         (None, 256)               0         
                                                                 
 dense_1 (Dense)             (None, 512)               131584    
                                                                 
 batch_normalization_6 (Batc  (None, 512)              2048      
 hNormalization)                                                 
                                                                 
 dropout_6 (Dropout)         (None, 512)               0         
                                                                 
 dense_2 (Dense)             (None, 4)                 2052      
                                                                 
=================================================================
Total params: 1,782,340
Trainable params: 1,778,116
Non-trainable params: 4,224
_________________________________________________________________
None



Epoch 1/20
119/119 [==============================] - 41s 316ms/step - loss: 1.7298 - accuracy: 0.2822 - val_loss: 1.5841 - val_accuracy: 0.3617 - lr: 0.0100
Epoch 2/20
119/119 [==============================] - 37s 309ms/step - loss: 1.5195 - accuracy: 0.2928 - val_loss: 2.0473 - val_accuracy: 0.2849 - lr: 0.0100
Epoch 3/20
119/119 [==============================] - 37s 311ms/step - loss: 1.4196 - accuracy: 0.3137 - val_loss: 1.3419 - val_accuracy: 0.3749 - lr: 0.0100
Epoch 4/20
119/119 [==============================] - 37s 308ms/step - loss: 1.3748 - accuracy: 0.3305 - val_loss: 1.4027 - val_accuracy: 0.3669 - lr: 0.0100
Epoch 5/20
119/119 [==============================] - 38s 316ms/step - loss: 1.3493 - accuracy: 0.3386 - val_loss: 1.3150 - val_accuracy: 0.3783 - lr: 0.0100
Epoch 6/20
119/119 [==============================] - 38s 315ms/step - loss: 1.3065 - accuracy: 0.3704 - val_loss: 1.2485 - val_accuracy: 0.4137 - lr: 0.0100
Epoch 7/20
119/119 [==============================] - 38s 315ms/step - loss: 1.2580 - accuracy: 0.4076 - val_loss: 1.2150 - val_accuracy: 0.4505 - lr: 0.0100
Epoch 8/20
119/119 [==============================] - 38s 315ms/step - loss: 1.1512 - accuracy: 0.4814 - val_loss: 1.1007 - val_accuracy: 0.5101 - lr: 0.0100
Epoch 9/20
119/119 [==============================] - 38s 315ms/step - loss: 1.0675 - accuracy: 0.5256 - val_loss: 1.0095 - val_accuracy: 0.5660 - lr: 0.0100
Epoch 10/20
119/119 [==============================] - 38s 315ms/step - loss: 0.9822 - accuracy: 0.5731 - val_loss: 1.0215 - val_accuracy: 0.5769 - lr: 0.0100
Epoch 11/20
119/119 [==============================] - 37s 311ms/step - loss: 0.9774 - accuracy: 0.5779 - val_loss: 1.3275 - val_accuracy: 0.4762 - lr: 0.0100
Epoch 12/20
119/119 [==============================] - 37s 310ms/step - loss: 0.9699 - accuracy: 0.5875 - val_loss: 1.1155 - val_accuracy: 0.5634 - lr: 0.0100
Epoch 13/20
119/119 [==============================] - 38s 315ms/step - loss: 0.8904 - accuracy: 0.6245 - val_loss: 0.8109 - val_accuracy: 0.6649 - lr: 0.0020
Epoch 14/20
119/119 [==============================] - 37s 314ms/step - loss: 0.8687 - accuracy: 0.6342 - val_loss: 0.7978 - val_accuracy: 0.6701 - lr: 0.0020
Epoch 15/20
119/119 [==============================] - 38s 316ms/step - loss: 0.8469 - accuracy: 0.6472 - val_loss: 0.7810 - val_accuracy: 0.6751 - lr: 0.0020
Epoch 16/20
119/119 [==============================] - 37s 313ms/step - loss: 0.8385 - accuracy: 0.6521 - val_loss: 1.3783 - val_accuracy: 0.3994 - lr: 0.0020
Epoch 17/20
119/119 [==============================] - 38s 316ms/step - loss: 0.8340 - accuracy: 0.6547 - val_loss: 0.7525 - val_accuracy: 0.6906 - lr: 0.0020
Epoch 18/20
119/119 [==============================] - 37s 310ms/step - loss: 0.8155 - accuracy: 0.6619 - val_loss: 0.8422 - val_accuracy: 0.6701 - lr: 0.0020
Epoch 19/20
119/119 [==============================] - 37s 310ms/step - loss: 0.8082 - accuracy: 0.6675 - val_loss: 0.7609 - val_accuracy: 0.6866 - lr: 0.0020
Epoch 20/20
119/119 [==============================] - 37s 313ms/step - loss: 0.7919 - accuracy: 0.6761 - val_loss: 0.7667 - val_accuracy: 0.6878 - lr: 0.0020



1/1 [==============================] - 0s 37ms/step - loss: 0.8769 - accuracy: 0.5938
1/1 [==============================] - 0s 195ms/step
Actual    : [0 1 3 0 1 3 2 1 0 2 0 3 2 2 3 0 3 2 0 2 1 3 2 2 0 0 3 0 1 2 0 2]
Prediction: [0 1 3 0 2 3 2 1 0 1 2 3 2 0 3 0 3 2 2 2 2 2 1 1 0 2 2 0 0 1 0 2]



              precision    recall  f1-score   support

       happy       0.78      0.70      0.74        10
     neutral       0.33      0.40      0.36         5
         sad       0.42      0.50      0.45        10
    surprise       1.00      0.71      0.83         7

    accuracy                           0.59        32
   macro avg       0.63      0.58      0.60        32
weighted avg       0.64      0.59      0.61        32





In [79]:
model_name = 'Model 6: Complex CNN'
save_object(history_complex_cnn, f'{MODEL_DIR}/{model_name} - history.pkl')
save_object(metric_complex_cnn, f'{MODEL_DIR}/{model_name} - metrics.pkl')
save_object(model_complex_cnn, f'{MODEL_DIR}/{model_name} - models.pkl')
In [80]:
loss_and_accuracy_comparisson(history_complex_cnn, data_type='Training Set')
loss_and_accuracy_comparisson(history_complex_cnn, data_type='Validation Set')
In [81]:
metric_complex_cnn.style.highlight_max(color = "lightgreen", axis = 0)
Out[81]:
  Accuracy Train Accuracy Val Accuracy Test
Model 6: Complex CNN (32) - GRAY 0.738699 0.747237 0.750000
Model 6: Complex CNN (64) - GRAY 0.728175 0.741009 0.687500
Model 6: Complex CNN (128) - GRAY 0.654709 0.690577 0.593750

⏩ Observations and Insights:

  • It can be observed that this complex model achieved the best performance accuracy 74.7%. This is probably because having a complex hierarchy enables the model to understand the data at different levels of abstraction, early layers learn basic features while deeper layers learn complex patterns of the data. However, the complex model needs more computational resources. As long as there are limited resources, the complexity of the model will be always a trade-off to negotiate.
  • In this case, this complex model takes advantage of working with Grayscale images encoded in 1 channel, instead of working with RGB images encoded in 3 channels (these RGB images will require even more resources).
  • Reviewing the learning curve, it shows a lot of variation in the validation set across epochs, meaning overfitting.
  • Reviewing the classification report, Happy and Surprise are the best-evaluated classes, and Sad and Neutral are the worst-evaluated classes. The best model "Model 6: Complex CNN (32) - GRAY" achieved 74.7% accuracy.

⏩ Recommendations:

  • Technique to deal with the overfitting needs to be applied to the model "Model 6: Complex CNN (32) - GRAY" and see if this improves the result in the validation set.
  • Explore other image augmentation techniques to provide more data to the model.
  • A review of the training dataset is necessary to confirm data are good enough.

Choosen Final Model¶

In [82]:
# Getting back the history and metrics result during trainning
with open(f'{MODEL_DIR}/Model 1: Base Neural Network - metrics.pkl', 'rb') as f: metric_base_cnn = pickle.load(f)
with open(f'{MODEL_DIR}/Model 2: Enhanced Neural Network - metrics.pkl', 'rb') as f: metric_enhanced_cnn = pickle.load(f)
with open(f'{MODEL_DIR}/Model 3: VGG16 - metrics.pkl', 'rb') as f: metric_vgg16 = pickle.load(f)
with open(f'{MODEL_DIR}/Model 4: Resnet V2 - metrics.pkl', 'rb') as f: metric_resnet = pickle.load(f)
with open(f'{MODEL_DIR}/Model 5: EfficientNet - metrics.pkl', 'rb') as f: metric_efficientnet = pickle.load(f)
with open(f'{MODEL_DIR}/Model 6: Complex CNN - metrics.pkl', 'rb') as f: metric_complex_cnn = pickle.load(f)
In [83]:
best_performed_models = pd.concat([
    metric_base_cnn.loc['Model 1: Base Neural Network (32) - RGB':'Model 1: Base Neural Network (32) - RGB'],
    metric_enhanced_cnn['Model 2: Enhanced Neural Network (64) - RGB':'Model 2: Enhanced Neural Network (64) - RGB'],
    metric_vgg16.loc['Model 3: Tuned VGG16 (32) - RGB':'Model 3: Tuned VGG16 (32) - RGB'],
    metric_resnet.loc['Model 4: Tuned Resnet V2 (32) - RGB':'Model 4: Tuned Resnet V2 (32) - RGB'],
    metric_efficientnet.loc['Model 5: EfficientNet (32) - RGB':'Model 5: EfficientNet (32) - RGB'],
    metric_complex_cnn.loc['Model 6: Complex CNN (32) - GRAY':'Model 6: Complex CNN (32) - GRAY']
])

best_performed_models
Out[83]:
Accuracy Train Accuracy Val Accuracy Test
Model 1: Base Neural Network (32) - RGB 0.705871 0.719108 0.78125
Model 2: Enhanced Neural Network (64) - RGB 0.826593 0.736789 0.68750
Model 3: Tuned VGG16 (32) - RGB 0.784367 0.712879 0.81250
Model 4: Tuned Resnet V2 (32) - RGB 0.513667 0.506731 0.37500
Model 5: EfficientNet (32) - RGB 0.263949 0.367892 0.31250
Model 6: Complex CNN (32) - GRAY 0.738699 0.747237 0.75000
In [84]:
_ = best_performed_models.plot(kind='bar', figsize=(20,3))
plt.xticks(rotation=0, fontsize=8)
plt.axhline(y=0.74, lw=1, ls='--', c='red')
plt.yticks(fontsize=6)
plt.title('Best Performed Models', fontsize=10)
plt.legend(fontsize=8)
plt.show()
In [85]:
# Getting back a model
# Preparing the environment
init_environment()
_, test_generator, _ = create_data_loaders(IMG_DIR, 'grayscale')
X_test, y_test = test_generator.next()

# Loading the model
model = model_builder_complex_cnn(input_shape=(48, 48, 1))
model.load_weights(f'{MODEL_DIR}/Model 6: Complex CNN (32) - GRAY.h5')

# Re-evaluate the model
loss, acc = model.evaluate(X_test, y_test, verbose=2)
model.evaluate(X_test, y_test, verbose=1)
print("Restored model, accuracy: {:5.2f}%".format(100 * acc))
Initializing TF Session and Seed

Data with color_mode=grayscale:
Found 15109 images belonging to 4 classes.
Found 127 images belonging to 4 classes.
Found 4977 images belonging to 4 classes.
Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d (Conv2D)             (None, 48, 48, 64)        320       
                                                                 
 batch_normalization (BatchN  (None, 48, 48, 64)       256       
 ormalization)                                                   
                                                                 
 leaky_re_lu (LeakyReLU)     (None, 48, 48, 64)        0         
                                                                 
 max_pooling2d (MaxPooling2D  (None, 24, 24, 64)       0         
 )                                                               
                                                                 
 dropout (Dropout)           (None, 24, 24, 64)        0         
                                                                 
 conv2d_1 (Conv2D)           (None, 24, 24, 128)       32896     
                                                                 
 batch_normalization_1 (Batc  (None, 24, 24, 128)      512       
 hNormalization)                                                 
                                                                 
 leaky_re_lu_1 (LeakyReLU)   (None, 24, 24, 128)       0         
                                                                 
 max_pooling2d_1 (MaxPooling  (None, 12, 12, 128)      0         
 2D)                                                             
                                                                 
 dropout_1 (Dropout)         (None, 12, 12, 128)       0         
                                                                 
 conv2d_2 (Conv2D)           (None, 12, 12, 512)       262656    
                                                                 
 batch_normalization_2 (Batc  (None, 12, 12, 512)      2048      
 hNormalization)                                                 
                                                                 
 leaky_re_lu_2 (LeakyReLU)   (None, 12, 12, 512)       0         
                                                                 
 max_pooling2d_2 (MaxPooling  (None, 6, 6, 512)        0         
 2D)                                                             
                                                                 
 dropout_2 (Dropout)         (None, 6, 6, 512)         0         
                                                                 
 conv2d_3 (Conv2D)           (None, 6, 6, 512)         1049088   
                                                                 
 batch_normalization_3 (Batc  (None, 6, 6, 512)        2048      
 hNormalization)                                                 
                                                                 
 leaky_re_lu_3 (LeakyReLU)   (None, 6, 6, 512)         0         
                                                                 
 max_pooling2d_3 (MaxPooling  (None, 3, 3, 512)        0         
 2D)                                                             
                                                                 
 dropout_3 (Dropout)         (None, 3, 3, 512)         0         
                                                                 
 conv2d_4 (Conv2D)           (None, 3, 3, 128)         262272    
                                                                 
 batch_normalization_4 (Batc  (None, 3, 3, 128)        512       
 hNormalization)                                                 
                                                                 
 leaky_re_lu_4 (LeakyReLU)   (None, 3, 3, 128)         0         
                                                                 
 max_pooling2d_4 (MaxPooling  (None, 1, 1, 128)        0         
 2D)                                                             
                                                                 
 dropout_4 (Dropout)         (None, 1, 1, 128)         0         
                                                                 
 flatten (Flatten)           (None, 128)               0         
                                                                 
 dense (Dense)               (None, 256)               33024     
                                                                 
 batch_normalization_5 (Batc  (None, 256)              1024      
 hNormalization)                                                 
                                                                 
 dropout_5 (Dropout)         (None, 256)               0         
                                                                 
 dense_1 (Dense)             (None, 512)               131584    
                                                                 
 batch_normalization_6 (Batc  (None, 512)              2048      
 hNormalization)                                                 
                                                                 
 dropout_6 (Dropout)         (None, 512)               0         
                                                                 
 dense_2 (Dense)             (None, 4)                 2052      
                                                                 
=================================================================
Total params: 1,782,340
Trainable params: 1,778,116
Non-trainable params: 4,224
_________________________________________________________________
None
1/1 - 0s - loss: 0.7360 - accuracy: 0.7500 - 495ms/epoch - 495ms/step
1/1 [==============================] - 0s 40ms/step - loss: 0.7360 - accuracy: 0.7500
Restored model, accuracy: 75.00%

Plotting the Confusion Matrix for the chosen final model¶

In [86]:
# Getting the history back as this a required parameter in our defined function.
with open(f'{MODEL_DIR}/Model 6: Complex CNN - history.pkl', 'rb') as f: history = pickle.load(f)

# Preparing the variables
y_pred = np.argmax(model.predict(X_test), axis=1)
y_test = np.argmax(y_test, axis=1)
print('Actual    :', y_test)
print('Prediction:', y_pred)
print('\n\n')

# Plotting the classification report and confussion matrix
results = metrics_score(y_test, y_pred, 'Model 6: Complex CNN (32) - GRAY', history['Model 6: Complex CNN (32) - GRAY'])
1/1 [==============================] - 0s 216ms/step
Actual    : [0 1 3 0 1 3 2 1 0 2 0 3 2 2 3 0 3 2 0 2 1 3 2 2 0 0 3 0 1 2 0 2]
Prediction: [0 1 3 0 2 3 2 1 0 1 1 3 2 0 3 0 3 2 0 2 1 3 1 2 0 2 1 0 1 1 0 2]



              precision    recall  f1-score   support

       happy       0.89      0.80      0.84        10
     neutral       0.44      0.80      0.57         5
         sad       0.75      0.60      0.67        10
    surprise       1.00      0.86      0.92         7

    accuracy                           0.75        32
   macro avg       0.77      0.76      0.75        32
weighted avg       0.80      0.75      0.76        32

⏩ Observations and Insights:

  • The selected model, with 33 layers, achieves 75% of accuracy in the final test.
  • The Happy and Surprise are the best evaluated classes.
  • The Neutral and Sad are the classes with the lowed f1-score.

⏩ Observations and Insights:

A couple of things can be done to try to improve this model:

  • Deal with the overfitting by tuning the already applied techniques (Dropout, BatchNormalization).
  • Explore kernel regularization as another possible technique to deal with overfitting.
  • Take a look at the dataset to confirm the features for the Neutral and Sad classes, verifying if they are representative.
  • Explore other image augmentation techniques to provide more data to the model.

Final Conclusions and Insights¶

⏩ Conclusions and Insights:

  • Complex models lead to having a better understanding of the data in classification problems. Overfitting is the highest risk in complex models.
  • Transfer models are an optimal approach to take advantage of models trained in millions of images, saving computation resources, however, overfitting is also present, and techniques to deal with this problem need to be considered/applied. Also, it is necessary to enable backpropagation in the final layers of the convolutional block to enable the model to learn the particularities of the problem and get the best of it.
  • The quality of the dataset is a key player in the modeling of a solution, bad poor data set leads to bad poor models.

Insights¶

Refined insights:¶

  • What are the most meaningful insights from the data relevant to the problem?

    ⏩ Even do, the dataset is balanced, features in some of the classes are difficult to identify, even for the human eyes. A review of the training dataset is necessary to confirm data is good enough. As it was mentioned before some images seem to belong to another class that the one they are currently on.

Comparison of various techniques and their relative performance:¶

  • How do different techniques perform? Which one is performing relatively better? Is there scope to improve the performance further?

    ⏩ With the exception of the "Model 4: Tuned Resnet V2" and "Model 5: Tuned EfficientNet", all models achieve more than 70% of accuracy. However, the time to tune and explore each model was huge.

    ⏩ "Model 4: Tuned Resnet V2" and "Model 5: Tuned EfficientNet" need to be recompiled with the "tune" parameter increased to enable a longer backpropagation, exploring if this improves the model.

Proposal for the final solution design:¶

  • What model do you propose to be adopted? Why is this the best solution to adopt?

    ⏩ There are 3 models in the final solution that can be selected:

    • "Model 2: Enhanced Neural Network (64) - RGB" that achieved 73.7% of accuracy.
    • "Model 3: Tuned VGG16 (32) - RGB" that achieved 71.3% of accuracy, had a consistent evolution through epochs during the training, and took advantage of the VGG16 models.
    • "Model 6: Complex CNN (32) - GRAY" that achieved 74.7% of accuracy, and had a good performance on the testing set.

    ⏩ With the current trained models and results, I recommend "Model 6: Complex CNN (32) - GRAY" as it achieved the best validation performance.

Final Submission¶


Executive Summary¶

⏩ In this classification problem, the goal was to find a model that could classify images in 4 different expressions: Happy, Surprise, Sad, and Neutral. For that, I have experimented with 21 different models and 3 approaches to try to find the best model that has an acceptable behavior in this classification task. Accordingly to the business, this task is quite important and is part of the roadmap they have in place for the present period. The main specification was to find a model that could classify the images better than a random guess, meaning that at least a 70% of accuracy would be acceptable for a proposed model.

⏩ There were 21 models that were trained and reviewed:

  1. 6 models related to a basic CNN model of 13 layers. These models were trained considering the batch sizes 32, 64, and 128, and the images set in RGB or Grayscale. The best result was obtained with the batch size 32 and the RGB set. These models included, in the convolutional layer block, the following layers: Conv2D, max_pooling2d (for downsampling the feature map), and dropout (for dealing with possible overfitting).
  2. 6 models related to an enhanced CNN model of 20 layers. These models were trained considering the batch sizes 32, 64, and 128, and the images set in RGB or Grayscale. The best result was obtained with the batch size 64 and the RGB set. These models were a little bit more complex than models 1, 2, and 3. They included "leaky_re_lu" (to ensure positive values always, an enhancement of the normal ReLu activation function), "batch_normalization" (to speed up training and use higher learning rates, making learning easier), and "max_pooling2d" (to downsample the feature map).
  3. 2 models using the transfer model approach with VGG16. This model considered the 18 layers of the convolutional VGG16 block and 7 layers added as part of the classification component. The first try saved/applied the already achieved weights of the VGG16 convolutional block, and in the second try, a tuned process took place, retraining the last 4 layers of the convolutional block (VGG16) with the training dataset belonging to the current classification problem. The best result was obtained with this tunned VGG16 transfer model.
  4. 2 models using the transfer model approach with ResNet V2. This model considered the 546 layers of the convolutional ResNet V2 block and 7 layers added as part of the classification component. The first try saved/applied the already achieved weights of the ResNet V2 convolutional block, and in the second try, a tunned process took place, retraining the last 5 layers of that convolutional block (ResNet V2) with the training dataset belonging to the current classification problem. The best result was obtained with this tuned ResNet V2 transfer model.
  5. 2 models using the transfer model approach with EfficientNet. This model includes the 561 layers of the convolutional EfficientNet block and 7 layers added as part of the classification component. The first try saved/applied the already achieved weights of the EfficientNet convolutional block, and in the second try, a tuned process took place, retraining the last 7 layers of that convolutional block (EfficientNet) with the training dataset belonging to the current classification problem. The best result was obtained with the no-tuned EfficientNet transfer model.
  6. 3 models related to a Complex CNN model of 33 layers and a batch size of 32, 64, and 128, considering only the images set in grayscale. These models were more complex, a kind of robust CNN model, They included "Conv2D", "leaky_re_lu" (to ensure positive values always, an enhancement of the normal ReLu activation function), "batch_normalization" (to speed up training and enable higher learning rates), "max_pooling2d" (to downsample the feature map), and dropout (to deal with possible overfitting). The best result was obtained with batch size 32.

⏩ The 3 followed approaches were:

  • A basic CNN model (Models from 1 to 6) that explored the use of the convolutional layers and how to deal with overfitting during the training.
  • A transfer model technique (Models from 7 to 12), to use the advantages of already training models and use those achieved weights to include them as part of the current classification problem.
  • A complex CNN model (Models from 13 to 15) that applies the learned lessons on the previous approach to finding a path that can achieve the expected result in the classification problem.

Findings

  • Complex models lead to having a better understanding of the data in classification problems. Overfitting is the highest risk in complex models. -Transfer models are an optimal approach to take advantage of models trained in millions of images, saving computation resources, however, overfitting is also present, and techniques to deal with this problem need to be considered/applied. Also, it is necessary to enable backpropagation in the final layers of the convolutional block to enable the model to learn the particularities of the problem and get the best of it.
  • The quality of the dataset is a key player in the modeling of a solution, bad poor data set leads to bad poor models.

Final Proposed Model

⏩ The model that achieved the greatest accuracy was the "Complex model with batch size 32 and focused on a grayscale images dataset". In only 20 epochs this model achieved approximately 75% of accuracy, looking promising for the future.

The selected and recommended model has 33 layers, the first 25 layers belong to the Feature extraction, and the last 8 layers to the Classification process. The feature extraction section is conformed of 5 convolutional blocks, each of them considered Conv2D, BatchNormalization, LeakyReLU, MaxPooling2D, and Dropout. The classification section related to the fully connected layers considered Flatten, Dense, BatchNormalization, and Dropout. Notice that Overfitting was considered and addressed in this complex robust CNN model.

Solution Summary¶

⏩ The "Complex model with batch size 32 and focused on a grayscale images dataset" is the selected and recommended model because:

  • Achieved the best result in all the 21 training models.
  • In only 20 epochs reached 75% of accuracy.
  • The learning curve shows that learning is still possible, so having more epochs to train will be beneficial.

⏩ The recommended model "Complex model with batch size 32 and focused on a grayscale images dataset" can be applied to the business to start classifying the images to address each image according to its class, this will improve the business operation by reducing time and cost. The company will not require hiring people for classification activities, allowing them to focus on their main objectives that take place after addressing each image in the right direction.

Benefits¶

⏩ Making a comparison between the cost and time invested in the classification of 30,000 images, we have:

⏩ For the Manual Process:

  • Invested time: 21 working days.
    20 sec. per image, for 30,000 images, one person will need 21 working days to complete the task.
  • Operational cost: 1,218.00 USD per month. No more tasks can be assigned.
    Considering the minimum hourly salary of 7.25 USD, the cost of having one person working on the classification of 30,000 images will cost 1,218 USD.

⏩ For the Deep Learning Supported Process

  • Invested time: 7 minutes.
    Approx 0.5 sec per batch of 32 images using Google Colab Pro Service. To classify 30,000 images, the model will need approx 7 minutes.
  • Operational cost: 9.99 USD per month. More tasks can be assigned.
    The Google Colab Pro Service costs 9.99 USD per month. This provides 100 compute units per month, Faster GPUs, and access to higher memory machines.

⏩ If the business implements this recommended Model, it will save time and money on this classification task, allowing them to focus resources on more strategic activities.

Recommendations for implementation¶

⏩ Next steps and recommendations

  • All previous models achieved better results with the RGB datasets, exploring the recommended model with the RGB dataset is advisable to verify if the model can improve.
  • The learning curve shows that the recommended model still learns at the end of the 20 epochs, it is necessary to explore if having more than 20 epochs gives a better result.

⏩ Main observations and risk

  • The image dataset needs to be reviewed and ensure acceptable quality.
  • The company needs also to explore the possibility to get more images, paying special attention to Neutral and Sad classes.

References¶

  • He, H., & Ma, Y. (Eds.). (2013). Imbalanced learning: foundations, algorithms, and applications.

  • Save, serialize, and export models | TensorFlow Core. (n.d.). TensorFlow. https://www.tensorflow.org/guide/keras/serialization_and_saving

  • Team, K. (n.d.). Keras documentation: Keras API reference. Keras API Reference. https://keras.io/api/

  • Module: tf | TensorFlow v2.13.0. (n.d.). TensorFlow. https://www.tensorflow.org/api_docs/python/tf

  • Colab Subscription Pricing. (n.d.). Colab Subscription Pricing. https://colab.research.google.com/signup